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U.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
SUBCOMMITTEE ON ENVIRONMENT 

HEARING CHARTER 


Ensuring Open Science at EPA 

Tuesday, February 11, 2014 
10:00 a.m. - 12:00 p.m. 

2318 Rayburn House Office Building 


PURPOSE 


The Subcommittee on Environment will hold a hearing entitled Ensuring Open Science at 
EPA on Tuesday, February 11, 2014 in Room 2318 of the Rayburn House Office Building. The 
purpose of this hearing is to examine options to improve the transparency and reproducibility of 
regulatory science used by the Environmental Protection Agency (EPA) and to receive testimony on 
Vac Secret Science Reform Act of 2014 (H.R. 4012), to prohibit EPA from proposing, finalizing, or 
disseminating regulations or assessments based upon scientific information unless such information 
is specifically identified and publically available in a manner sufficient for independent analysis and 
reproducibility. 

WITNESS LIST 


• The Honorable John Graham, Dean, School of Public and Environmental Affairs, Indiana 
University 

• Dr. Louis Anthony Cox, Jr., Chief Sciences Officer, Next Health Technologies, Clinical 
Professor, Biostatistics and Informatics, Colorado Health Sciences Center, and President, 
Cox Associates 

• Dr. Ellen Silbergeld, Professor, Bloomberg School of Public Health, Johns Hopkins 
University 

• Mr. Raymond Keating, Chief Economist, Small Business & Entrepreneurship Council 


BACKGROUND 

Science has been central to EPA’s mission and functions since its establishment in 1970. The 
Agency’s recent Scientific Integrity Policy describes science as “the backbone of the EPA’s 
decision-making.”’ Efforts to encourage and guarantee open scientific research and assessment at 
the Environmental Protection Agency are based in a number of historical, legal, and 
administrative origins. 


' hltD: //www .eo a.ttov/osa'Ddfs/eDa scientific inleeritv policy 201201 IS.pdf . 
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In 1983, then-Administrator William Ruckelshaus wrote a memo to all EPA employees 
dictating that the agency should operate as though it were “in a fishbowl.” The memo stressed 
the importance of being as open as possible, while also providing the fuUest possible public 
participation in decision-making.^ EPA Administrator Gina McCarthy echoed this priority in her 
confirmation hearing, stating; “The rule of law, along with sound science and transparency, is 
one of EPA’s core values and, iff am confirmed, it will continue to guide all EPA actions.”® 
Similarly, she stated that, “EPA is committed to transparency with regard to the scientific bases 
of agency decision-making.”^ The importance of science to EPA’s regulatory decisions is a 
critical component of several environmental laws, including the Environmental Research, 
Development, and Demonstration Authorization Act, the Clean Air Act, the Clean Water Act, 
and the Safe Drinking Water Act. 

Recent EPA and White House scientific integrity, regulatory, and open access policies 
indicate further support for open science. Executive Order 13563 requires that regulations “be 
based upon the best available science.”* Similarly, President Obama’s March 2009 Scientific 
Integrity Memo states that “[t]o the extent permitted by law, there should be transparency in the 
preparation, identification, and use of scientific and technological information in 
policymaking.”* 

Following up on this direction, the White House Office of Science and Technology Policy 
(OSTP) Memo from December 2010 states that, “agencies should expand and promote access to 
scientific information by making it available online in open formats. Where appropriate, this 
should include data and models underlying regulatory proposals and policy decisions.”’ OSTP 
also issued a Memorandum last year on “Increasing Access to the results of Federally Funded 
Scientific Research,” in which the President’s Science Advisor, John Holdren, explained: “The 
Adminishation is committed to ensuring that, to the greatest extent and with the fewest 
constraints possible... the direct results of federally funded scientific research are made available 
to and useful for the public, industry, and the scientific community. . . Such results include peer- 
reviewed publications and digital data.”® 

In order to provide Agency-specific guidelines emanating from the Administration’s 
Scientific Integrity Memos, EPA’s Scientific Integrity Policy issued in 2012 states: “Scientific 
research and analysis comprise the foundation of all major EPA policy decisions. Therefore, the 
Agency should maintain vigilance toward ensuring that scientific research and results are 
presented openly and with integrity, accuracy, timeliness, and the full public scmtiny demanded 
when developing sound, high-quality envirorunental science.”® 


^ htt p://vvww2.cpa.gov/ahoiitepatnickelshaus-takes-steps-imprQve-flow-agencv-uifonnatiQn-fishbowi-poh'cv#memo . 
’ hltD://www.eDw.senate.gOY/Dublic/index.cfiii?FuseAct ion= Hearings.Hearing&Hearing id=d71fd4b6-ce77-3a9S- 
46aO-fbQ2bOcaeOed 
Ibid. 

^ http://www,gDO.gov/fdsv s /pkg/FR-201 l-01-21/Ddf/201 l-1385.pdf 

‘ littp://www.\vhitehouse.Eov/Hie-Dress-ofrice/nieiiioranduin-head!i-6xecutive-deDarbiients-and-agencies-3-9-09 
’ hnp:/Avww.whitehouse.gov/site.'i/defaalt/files/tnicrosites/o!itD/scientific-inlegi'itv-inemo-12l7 2 010.Ddf . 

^ htti3://www.whitehouse.gov/sites/deFault/Filcs/iTiicr[)sites/ostp/osb> public access memo 20i3.p df. 

’ http://www.epa, gov/osa/pdfs/epa s cientific mtegiitv policy 2Qi20n5.pdf . 


2 



5 


Developed in response to Office of Management and Budget (OMB) guidelines issued 
following provisions of the Treasury and General Government Appropriations Act for Fiscal 
Year 2001 (Public Law 106-554; H.R. 5658), EPA’s Guidelines for Ensuring and Maximizing 
the Quality, Objectivity, Utility, and Integrity, of Information Disseminated by the Environmental 
Protection Agency state that the Agency is “committed to providing public access to 
environmental information” and that, in order to fulfill its mission, “EPA must rely upon 
information of appropriate quality for each decision we make.” EPA also notes the limitations of 
these guidelines, stating that they “provide non-binding policy and procedural guidance, and are 
therefore not intended to create legal rights, impose legally binding requirements or obligations 
on EPA or the public when applied in particular situations, or change or impact the status of 
information we disseminate, nor to contravene any other legal requirements that ifaay apply to 
particular agency determinations or other actions.”'® 

OMB Circular A-1 10 also indicates that the federal government has a right to data produced 
under certain federally-funded research awards. In 1999, following an amendment to the 
Omnibus Appropriations Act for FY 1999 (often referred to as the “Shelby Amendment” as the 
amendment was sponsored by Senator Richaift Shelby) OMB revised this circular to “ensure that 
all data produced under an award will be made available to the public through the procedures 
established under the Freedom of Information Act.”" 

ADDITIONAL REAPING 


• Bipartisan Policy Center, Science for Policy Project, Improving the Use of Science in 
Regulatory Policy. August 2009. Available at; 

httD://bipartisanpolicv.org/sites/'default/files/BPC%20Science%20Report%20fnl.t)df 

• Office of Science and Technology PoUcy, Memorandum for the Heads of Executive 
Departments and Agencies, February 2013. Available at: 

http://www.whitehouse.gov/sites/default/files/microsites/ostp/ostD public access memo 
2013.pdf 

• Committee on Science, Space, and Technology, Strengthening Transparency and 
Accountability within the Environmental Protection Agency, November 1 4, 20 13. 
Available at: hltp://science.house.gov/hearing/fiill-committee-hearmg-strencthening- 
transparencv-and-accountabilitv-within-environmental 

• Committee on Science, Space, and Technology, Scientific Integrity & Transparency, 
March 5, 2013. Available at: httD://science.house.gov/hearing/subcommittee-research- 
scientific-integritv-transparencv 

• Committee on Science, Space, and Technology, Fostering Quality Science at EPA: 
Perspectives on Common Sense Reform - Day II, February 3, 2012. Available at: 
http://science.house.gov/hearing/energv-and-environment-subcommittee-hearing- 
fostering-aualitv-science-epa-perspectives-0 


” httD://www,eaa.gov/oualitv/infomiationeuidelines/docunients/EPA InfoOualitvGuidelines.pdf . 
“ http://www, whitehou.se.gov/sites/defaulFfilcs/onib/fedree/al 10- fi jialnotice.htn il 
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• Committee on Science, Space, and Technology, EFA 's Impact on Jobs and Energy 
Affordability: Understanding the Real Costs and Benefits of Environmental Regulations, 
June 6, 2012. Available at: httD://science.house.gov/hearing/subcommittee-energv-and- 
enviromnent-hearing-etia%E2%80%99s-imDact-iobs-and-energv-affordabilitv 
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2bs™ H. K. 4U1Z 

To proliibit tbe Environmental Pi-oteetion Agency from proposiug, finalizhig, 
oi' disseminating regulations or assessments based upon science tliat 
is not transparent or reproducible. 


IN THE HOUSE OP REPRESENTATIVES 
Pebruaey 6, 2014 

Ml’. SCHWEIICEET (for bimself, Mi’. Smitu of Texas, Mr, IIatj,, Mr. Bboun 
of Georgia, Mi’. CuuiBliSON, Mr. Brtdenstinb, Mrs. LmrMia, Mr. 
Roheabaghbr, Mr. CoijIjIns of New York, Mr. Btogbss, Mi’. Olson, 
Mr. Cramer, Mi-. Bucshon, Mr. IIttijTGEen, Mi’. Neugebaubr, Mi’. 
Palazzo, Mi’. Brooks of Alabama, Mi’. Salmon, and Mi’. Franks of 
Arizona) intixidueed the foUowing bill; whicli was refen’ed to the Com- 
mittee on Science, Space, and Teclinologj'' 


A BILL 

To proliibit the Environmental Protection Agency from pro- 
posing, finalizing, or disseminating regulations or assess- 
ments based upon science tliat is not transparent or 
reproducible. 

1 Be it mactecl by the Senate and Home of Rejoresenta- 

2 tives of the United States of America in Congress assembled, 

3 SECTION 1. SHORT TITLE. 

4 This Act may be cited as the “Secret Science Reform 


5 Act of 2014". 
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2 

1 SEC. 2. DATA TKANSPARENCy. 

2 Section 6(b) of the Emdromiiental Kesearch, Devel- 

3 opnient, and Demonstration Authorization Act of 1978 

4 (42 U.S.C. 4363 note) is amended to read as follows; 

5 “(b)(1) The Administi-ator shall not propose, finalize, 

6 or disseminate a covei-ed action unless all scientific and 

7 teclmical infoi-mation relied on to support such covered ac- 

8 tion is — 

9 “(A) specifically identified; and 

10 “(B) publicly available in a manner that is suf- 

11 ficient for independent analysis and substantial re- 

12 production of research results. 

13 “(2) Nothing m the subsection shall be constmed as 

14 requii’ing the public dissemination of information the dis- 

15 closure of which is prohibited by law. 

16 . “(3) In this subsection — 

17 “(A) the term ‘covered action’ means a risk, ex- 

18 posm-e, or hazard assessment, criteria document, 

19 standard, limitation, regrdation, regulatory impact 

20 analj’sis, or guidance; and 

21 “(B) the tenn ‘scientific and teclmical infonna- 

22 tion’ includes — 

23 “(i) materials, data, and associated pi-oto- 

24 cols necessary to understand, assess, and ex- 

25 tend conclusions; 


•HR 4012 m 
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3 

1 “(ii) computer codes and models involved 

2 in the creation and analj'sis of such infoiana- 

3 tion; 

4 “(iii) recorded factual materials; and 

5 “(ivj detailed descriptions of how to access 

6 and use such information.”. 

O 


•HK 4012 m 
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Chairman ScHWEiKERT. Let’s have at it. The Subcommittee on 
Environment will come to order. And that was my gavel. Welcome 
to today’s hearing “Ensuring Open Science at EPA.” In front of you 
are packets containing the written testimony, biographies, and 
truth-in-testimony disclosures for today’s witnesses. 

This is also my very first hearing chairing this Subcommittee 
and so I was going to do something a little off the normal script. 
I first was going to turn to my soon-to-be very good friend from Or- 
egon, Ranking Member Bonamici, and I have got to say thank you 
for your kindness. I am going to take a somewhat different tack 
than often happens at many of these. I actually have a fascination 
with the underlying science and want to try to do this as credibly 
as possible, you know, because this in some ways should be almost 
beyond the — sometimes the right/left paradigm we engage in. This 
is hopefully about facts. 

On a philosophical level — and forgive me for going there — for to- 
day’s hearing how do you have a civil society with our public when 
our leaders, when the people around us, almost no one trusts our 
institutions anymore? You know, how do you have a society and 
hold it together when we don’t trust our government, we don’t trust 
our agencies, we don’t trust so much around us? And my great 
hope is this being sort of a first step is a movement towards a level 
of transparency where I don’t care whether you are a group from 
the right, left, or just someone from academia. The ability in to- 
day’s world — when my laptop computer is now more powerful than 
the quad Xeon server I have at home, for all of us, our ability to 
actually — if you were crazy enough or were interested enough in 
your quant class to take the data, to understand it, to analyze it, 
to have an opinion, does that openness, does that transparency — 
and the President actually talked about this when he was first 
elected, that a transparent, open government develops hopefully a 
faith and trust with its population. Can we head that direction? 

And I know we get into certain things like I consider sort of red 
herrings, absurdities. There are ways to protect people’s privacy. 
We do it every day. I come more from the financial side of the 
world having sat on Financial Services before, and data that was 
collected by CFPB and so many of the other agencies, they have 
systematic methodologies where they protect individuals’ privacy. 

But I am — my great hope here as we sort of move forward on 
H.R. 4012, that we are sort of building a precedence of how do I 
build public data for public policy and public policy by sort of egali- 
tarian public data where we all have the right to know what is un- 
derlying? 

And my last caveat for — and I have been thinking about this one 
a lot — and this is both for my friends on the right, the left, and our 
staff, you are going to have to step beyond sort of the confirmation 
bias. Let’s say we are here a couple years from now and all of us 
have access to underlying baseline data and it is being used for 
regulator or policy, don’t think it is always going to say what you 
think it is going to say. There are going to be times when the data 
sets may say the agency isn’t going far enough. There may be other 
times it turns around and says when you stress the data that we 
need to be going a very different approach, but at least it will be 
honest. And being fixated on sort of crowdsourcing of information. 
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I believe the crowd does purify policy and I hope we are going that 
direction. 

[The prepared statement of Mr. Schweikert follows:] 

Prepared Statement of Subcommittee on Environment 
Chairman David Schweikert 

I actually have fascination with the underlying science. And I want to try and do 
this as credibly as possible. Because this, in some ways this should be almost be- 
yond the sometimes left right paradigm we engage in, this should be about facts. 

How do you have a civil society when our leaders, our public don’t trust our insti- 
tutions anymore? We don’t trust our government, we don’t trust our agencies, we 
don’t trust so much around us. A transparent, open government develops, hopefully, 
a faith with its population. 

My great hope is, this being sort of a first step, is movement towards a level of 
transparency, where I don’t care if you a group from the right or the left, or just 
someone from academia the ability in today’s world ... to take the data, to under- 
stand it, to have an opinion. 

A transparent, open government develops, hopefully, a faith with its population. 
Can we head that direction?There are ways to protect people’s privacy, we do it 
every day. 

My great hope here is we sort of move forward on H.R. 4012, that we are sort 
of building a precedence on how do I build public data for public policy, and public 
policy by egalitarian public data, where we all have the right to know what is un- 
derlying. 

You are going to have to steep beyond confirmation bias. Don’t think it (the data) 
is always going to say what you think it is going to say. There is bound to be times 
when the data sets may say the agency isn’t going far enough. There may be other 
times it says when you stress the data, we need to be going with a different ap- 
proach. 

I believe the crowd does purify policy, and I hope we are going in that direction. 

Chairman Schweikert. And for an opening statement, my 
friend, Ms. Bonamici, Ranking Member. 

Ms. Bonamici. Thank you very much. Chairman Schweikert. 

I want to start by saying welcome and to offer my sincere con- 
gratulations to you on becoming our new Subcommittee Chairman. 
I am looking forward to working with you. I am hopeful that we 
can find common ground and develop meaningful solutions to our 
Nation’s important environmental challenges. 

And, I agree; improving transparency and public access, espe- 
cially to federally funded research at the EPA or at any federal 
agency, is an important objective and one that I fully support. And 
although there may be disagreements about — among the Sub- 
committee Members about various actions that the EPA may be 
considering, I am confident that we all support increased trans- 
parency. 

Unfortunately, it appears that the language in the bill we are 
discussing today called the Secret Science Reform Act may actually 
prohibit EPA from increasing transparency. And I hope that this 
isn’t an attempt to prevent or impede the EPA from promulgating 
regulations and performing its congressionally mandated priority 
objective of protecting human health and the environment. 

If implemented as written, this bill would actually prevent the 
EPA from using the best available science to inform its regulatory 
actions. The EPA relies on thousands of peer-reviewed articles as 
part of their scientific review, and under this proposal, if for any 
reason all of the scientific and technical information associated 
with those articles was not publicly available, the EPA would have 
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to proceed as if those studies did not exist. And that is not in the 
best interest of the American people who are our constituents. It 
is also not clear whether this proposal is retroactive. If so, then the 
legislation would essentially nullify all the progress we have made 
to date to improve the quality of the air our children and in fact 
all of our constituents breathe and the water that they drink. 

I am also concerned about the potential negative impacts that 
the bill could have on the scientific community. Researchers and 
organizations may be hesitant to conduct EPA-funded research if 
they are required to disclose protected information like health 
records. Historically, researchers have been able to assure individ- 
uals participating in their studies that their personal information 
is safe, and that helps attract participants. 

Now, last year, this Committee took the unusual action of issuing 
a subpoena to acquire data that the EPA relied on when developing 
air quality regulations. This data, the basis of the Harvard 6 stud- 
ies and the American Cancer Society study, contains personal 
health records of hundreds of thousands of Americans. And I pre- 
sume that this is an example of the so-called secret science that in- 
spired this bill. But contrary to the assertion that the science be- 
hind those studies is secret, in fact, the legal owners of the data 
sets. Harvard University and the American Cancer Society, do 
allow legitimate researchers access to this information and they 
have procedures in place to protect it. 

So it is interesting this Committee did spend a significant 
amount of time scrutinizing HealthCare.gov and claiming that the 
website actually puts personal health records at risk. Frankly, I am 
a bit surprised that my colleagues do not now recognize the impor- 
tance of protecting studies that actually do contain personal health 
information. 

But perhaps what is more troubling about this proposal — and I 
look forward to discussing it — is that it ignores the good work al- 
ready done by this Committee. In 2010, this Committee reauthor- 
ized the America COMPETES Act, which requires the Office of 
Science and Technology Policy or OSTP to issue guidance to all fed- 
eral agencies on the development of clear and coordinated policies 
to increase access to federally funded published research and dig- 
ital scientific data. And it is my understanding that the EPA is cur- 
rently in the process of developing policies pursuant to that guid- 
ance. 

This bill also seems to be inconsistent with the data and public 
access provisions included in the majority’s FIRST Act. Although 
there are some open questions about specific provisions of the 
FIRST Act, the bill takes the more appropriate government-wide 
approach and requires consultation and input from the scientific 
and stakeholder community. It is worth having a real discussion — 
and again, we look forward to that — about how we can improve 
transparency and data access across the federal government. 

Additionally, as we have discussed, I hope we are able to have 
another hearing on this issue. I strongly encourage the participa- 
tion of the EPA so that the Agency has an opportunity to appear 
before the Committee and provide on the record their analysis 
about the provisions of this bill. It would be logical — and I suggest 
this, Mr. Chairman — that we hold such a hearing in conjunction 
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with our colleagues on the Research and Technology Subcommittee 
because they are also examining this issue very closely. 

Mr. Chairman, I truly hope we can work together to find a way 
to improve public access to federally funded research in a manner 
that does not compromise the EPA’s mission to protect human 
health and the environment. 

Thank you again, and welcome to the Committee, Mr. Chairman, 
and I yield back the balance of my time. 

[The prepared statement of Ms. Bonamici follows:] 

Prepared Statement of Subcommittee on Environment 
Ranking Minority Member Suzanne Bonamici 

Thank you, Chairman Schweikert. I would like to start by saying welcome, and 
I offer sincere congratulations to you on becoming our new Subcommittee Chairman. 
I am looking forward to working with you and am hopeful that we can find common 
ground and develop meaningful solutions to our nation’s important environmental 
challenges. 

Improving transparency and public access to federally funded research at EPA, 
or at any federal agency, is an important objective and one that I fully support. Al- 
though there may be disagreements among the Subcommittee Members about var- 
ious actions that the EPA may be considering, I am confident that we all support 
increased transparency. 

Unfortunately, it appears the language in the bill we are discussing today, called 
the “Secret Science Reform Act,” may actually prohibit EPA from increasing trans- 
parency. I hope that this is not an attempt to prevent or impede the EPA from pro- 
mulgating regulations and performing its Congressionally-mandated priority objec- 
tive of protecting human health and the environment. 

If implemented, this bill would actually prevent the EPA from using the best 
available science to inform its regulatory actions. EPA relies on thousands of peer- 
reviewed articles as part of their scientific review. Under this proposal, if for any 
reason all of the scientific and technical information associated with those articles 
was not publicly available, EPA would have to proceed as if those studies did not 
exist. That is not in the best interest of the American people - our constituents. 

It is also not clear whether this proposal is retroactive. If so, then the legislation 
would essentially nullify all the progress we’ve made to date to improve the quality 
of the air our children — and all of our constituents for that matter — breathe and the 
water they drink. 

I am also concerned about the potential negative impacts that the bill could have 
on the scientific community. Researchers and organizations may be hesitant to con- 
duct EPA-funded research if they are required to disclose protected information like 
health records. Historically, researchers have been able to assure individuals par- 
ticipating in their studies that their personal information is safe, and that helps at- 
tract participants. 

Last year, this Committee took the unusual action of issuing a subpoena to ac- 
quire data that the EPA relied on when developing air quality regulations. This 
data, the basis of the Harvard Six Cities study and the American Cancer Society 
study, contains the personal health records of hundreds of thousands of Americans. 
I presume that this is an example of the so-called “secret science” that inspired this 
bill. 

But contrary to the assertion that the science behind those studies is “secret, in 
fact the legal owners of these data sets. Harvard University and the American Can- 
cer Society, allow legitimate researchers access to this information and have proce- 
dures in place to protect it. 

It’s interesting — this Committee spent a significant amount of time scrutinizing 
Healthcare.gov and claiming that the website puts personal health records of mil- 
lions at risk; frankly I am a bit surprised that my colleagues do not now recognize 
the importance of protecting studies that actually do contain personal health infor- 
mation. 

But perhaps what is more troubling about this proposal is that it ignores the good 
work already done by this Committee. In 2010, this Committee reauthorized the 
America COMPETES Act, which requires the Office of Science and Technology Pol- 
icy (OSTP) to issue guidance to all federal agencies on the development of clear and 
coordinated policies to increase access to federally funded published research and 



14 


digital scientific data. It’s my understanding that the EPA is in the process of devel- 
oping policies pursuant to this guidance. 

This bill also seems to be inconsistent with the data and public access provisions 
included in the Majority’s FIRST Act. Although there are some open questions about 
specific provisions of the FIRST Act, the bill takes the more appropriate govern- 
ment-wide approach and requires consultation and input from the scientific and 
stakeholder community. 

It is worth having a real discussion about how we can improve transparency and 
data access across the federal government. Additionally, as we have discussed, I 
hope we are able to have another hearing on this issue. I strongly encourage the 
participation of the EPA so that the agency has an opportunity to appear before the 
Committee and provide — on the record — their analysis about the provisions of this 
bill. It would be logical to hold such a hearing in conjunction with our colleagues 
on the Research and Technology Subcommittee because they have been examining 
this issue closely. 

Mr. Chairman, I hope that we can work together to find a way to improve public 
access to federally funded research in a manner that does not compromise the EPA’s 
mission to protect human health and the environment. 

Thank you, Mr. Chairman and I 3deld back. 

Chairman SCHWEIKERT. Thank you, Ranking Member Bonamici. 

Now, I would like to turn to the Chairman of the full Committee, 
the gentleman — and I emphasize gentleman — from Texas, Mr. 
Smith, opening statement, please. 

Chairman Smith. Thank you, Mr. Chairman, and congratulations 
on chairing your first Subcommittee hearing. 

The Secret Science Reform Act of 2014 is a result of more than 
two years of investigative work on the part of the Science, Space, 
and Technology Committee. This work was initiated when the En- 
vironmental Protection Agency failed to live up to its public com- 
mitment to make the data that supports its most costly air regula- 
tions available to the public. In September 2011, then-Assistant 
Administrator Gina McCarthy committed to provide this Com- 
mittee with the data EPA relied upon to justify its claims about air 
quality and health effects. In 2012, the President’s own science ad- 
visor John Holdren testified that “absolutely the data on which reg- 
ulatory decisions are based should be made available to the Com- 
mittee and should be made public.” 

The Committee sought this data for a simple reason: to see 
whether the science supports EPA’s rules. An open and transparent 
government requires its disclosure. Through this process, we 
learned that much of the data either no longer exists or was never 
in the Agency’s possession. Not only are EPA’s claims not inde- 
pendently verifiable, the Agency cannot provide evidence to justify 
them. As a result, the American people have no way of knowing the 
truth. 

The EPA’s mission is to protect public health and the environ- 
ment, but the Agency’s regulations impact all aspects of our econ- 
omy. Sound public policy requires precise decision-making that 
properly balances competing needs. While the Agency is charged 
with setting standards that are “requisite to protect public health,” 
those standards should be no more restrictive than necessary. 
Transparency and independent verification are basic tenets of 
science and must inform sound environmental policy. When the 
EPA does not follow these basic steps, it fails in its obligation to 
the American people and raises suspicions about whether its regu- 
lations can be justified. 
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It is unfortunate that our Nation’s environmental policy has be- 
come one of the most contentious issues in Washington, but a dis- 
cussion about the merits of any particular regulation is meaning- 
less if the public cannot trust the underlying science, and that is 
impossible if the information isn’t even available. 

Everyone agrees that we need to protect the environment, but it 
should be done in a way that is transparent and honest. This bill 
encourages those principles. The Secret Science Reform Act of 2014 
has two basic elements. One, it prohibits EPA from issuing regula- 
tions unless all scientific and technical information relied upon is 
specifically identified; and two, it requires that information to be 
publicly available in a manner that is sufficient for independent 
analysis and reproduction of research results. 

Americans impacted by EPA regulations have a right to see the 
data and determine for themselves if the Agency’s actions are 
based on sound science or a partisan agenda. This bill ensures 
transparency and accountability. It is hard to imagine a single rea- 
son why anyone would oppose this basic principle that is consistent 
with the Administration’s policies on transparency. James Madison 
may have explained this best when he said that “a popular govern- 
ment without popular information or the means of acquiring it is 
but a prologue to a farce or a tragedy, or perhaps both. Knowledge 
will forever govern ignorance, and a people who need to be their 
own governors must arm themselves with the power knowledge 
gives.” 

Given the EPA’s aggressive agenda and its willingness to play 
fast and loose with the law, the Agency should be forced to live up 
to the claims of transparency it so readily espouses. The American 
people deserve the facts and so does good policy. 

Thank you, Mr. Chairman. I yield back. 

[The prepared statement of Mr. Smith follows:] 

Prepared Statement of Full Committee Chairman Lamar S. Smith 

The Secret Science Reform Act of 2014 is the result of more than two years of 
investigative work on the part of the Science, Space, and Technology Committee. 
This work was initiated when the Environmental Protection Agency (EPA) failed to 
live up to its public commitment to make the data that supports its most costly air 
regulations available to the public. 

In September 2011, then- Assistant Administrator Gina McCarthy committed to 
provide this Committee with the data EPA relied upon to justify its claims about 
air quality and health effects. In 2012, the President’s Science Advisor, John 
Holdren, testified that, “Absolutely, the data on which regulatory decisions.are 
based should be made available to the Committee and should be made public.” 

The Committee sought this data for a simple reason: to see whether the science 
supports EPA’s rules. An open and transparent government requires its disclosure. 
Through this process, we learned that much of the data either no longer exists or 
was never in the agency’s possession. Not only are EPA’s claims not independently 
verifiable, the agency cannot provide evidence to justify them. 

As a result, the American people have no way of knowing the truth. EPA’s mis- 
sion is to protect public health and the environment. But the agency’s regulations 
impact all aspects of our economy. 

Sound public policy requires precise decision-making that properly balances com- 
peting needs. While the agency is charged with setting standards that are “requisite 
to protect public health,” those standards should be no more restrictive than nec- 
essary. 

Transparency and independent verification are basic tenants of science and must 
inform sound environmental policy. When the EPA does not follow these basic steps. 
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it fails in its obligation to the American people and raises suspicions about whether 
its regulations can be justified. 

It’s unfortunate that our nation’s environmental policy has become one of the 
most contentious issues in Washington. But a discussion about the merits of any 
particular regulation is meaningless if the public cannot trust the underlying 
science. And that’s impossible if the information isn’t even available. 

Everyone agrees that we need to protect the environment. But it should be done 
in a way that is transparent and honest. This bill encourages those principles. 

The Secret Science Reform Act of 2014 has two basic elements: 

1) It prohibits EPA from issuing regulations unless all scientific and technical 

information relied upon is specifically identified; and, 

2) It requires that information to be publicly available in a manner that is suffi- 
cient for independent analysis and reproduction of research results. 

Americans impacted by EPA regulations have a right to see the data and deter- 
mine for themselves if the agency’s actions are based on sound science or a partisan 
agenda. This bill ensures transparency and accountability. 

It’s hard to imagine a single reason why anyone would oppose this basic principle 
that is consistent with the Administration’s policies on transparency. 

James Madison may have explained this best when he said that, “A popular gov- 
ernment without popular information or the means of acquiring it, is but a Prologue 
to a Farce or a Tragedy — or perhaps both. Knowledge will forever govern ignorance, 
and a people who mean to be their own Governors must arm themselves with the 
power knowledge gives.” 

Given the EPA’s aggressive agenda and its willingness to play fast and loose with 
the law, the agency should be forced to live up to the claims of transparency it so 
readily espouses. The American people deserve the facts. And so does good policy. 

Chairman SCHWEIKERT. Thank you, Chairman Smith. 

I now want to recognize my other bookend from Texas, the Rank- 
ing Member of the full Committee, Ms. Johnson. 

Ms. Johnson. Thank you very much, Mr. Chairman. And I 
would like to echo Ms. Bonamici in congratulating you on being 
named the Chair of the Subcommittee and look forward to working 
with you in this capacity and have been impressed with your par- 
ticular statements, this meeting and others. 

Unfortunately, I regret that today’s hearing might be a rough 
start in that regard. That is because the Secret Science Reform Act 
of 2014 continues to be one of the most regrettable sagas in the his- 
tory of this esteemed Committee. Out of all the years I have served 
on this Committee, this term has been the worst experience. 

This saga began in the last Congress with majority requests for 
data associated with studies that the EPA relied on for certain 
clean air regulations. It continued in August of last year when the 
Chairman issued the first subpoena from this Committee in over 
20 years to obtain that same data. And now, we are here today to 
discuss this misguided and mislabeled legislation. 

I want to be clear. The Secret Science Reform Act of 2014 is built 
on a false premise. None of the science that have been in question 
during this two-year affair is secret. Is the data protected? Of 
course it is. The data contains the personally identifiable health in- 
formation of hundreds of thousands of American citizens. Nonethe- 
less, as the Democratic minority has repeatedly pointed out, legiti- 
mate researchers do have access to this data. So what is the prob- 
lem? What legitimate researchers cannot already access this data? 

At the August 1, 2013, meeting to authorize a subpoena, the 
Chairman indicated Dr. James Enstrom could not access the Amer- 
ican Cancer Society data. As I have pointed out before. Dr. 
Enstrom has a long history of conducting research and performing 
consultant work for the tobacco industry. 
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Now, that brings us to today’s hearing. Mr. Chairman, all three 
of the majority witnesses also have significant ties to the tobacco 
industry. First, we have Dr. John Graham. While he headed the 
Harvard Center for Risk Analysis, he personally solicited research 
funding from Philip Morris. Moreover, he invited Philip Morris 
public relation officials to review a draft chapter of his book on the 
subject of secondhand smoke. Dr. Graham’s center ultimately re- 
ceived tens of thousands of dollars on grants from Philip Morris’ 
subsidiary, Kraft General Foods. 

Next, we have Dr. Tony Cox, who has received numerous re- 
search grants from Philip Morris tobacco and has collaborated on 
research with internal Philip Morris scientists. In addition. Dr. Cox 
has served as a litigation consultant for the Philip Morris and R.J. 
Reynolds tobacco companies. 

Finally, we have Dr. Ray Keating. Dr. Keating’s organization, the 
Small Business & Entrepreneurship Council, and its predecessor, 
the Small Business Survival Foundation, has solicited and received 
funding from tobacco companies. Moreover, documentation seems 
to suggest a large amount of collaboration with tobacco companies. 
For instance, in the mid-’90s. Dr. Keating released a series of re- 
ports of FDA tobacco regulations and their negative effects on 
small business and also filed comments with the FDA on the same 
topic. These reports relied upon a study commissioned by Dr. 
Keating’s organization and conducted by the American Economics 
Group. What Dr. Keating didn’t mention in his reports or FDA 
comments is that the Small Business Survival Foundation was act- 
ing as a go-between for the tobacco industry. Tobacco companies’ 
emails show that the study in question was jointly funded and or- 
ganized by Philip Morris and R.J. Reynolds tobacco. 

The reason I highlight this, Mr. Chairman, is that EPA is a pub- 
lic health agency. I am a health professional. I find it deeply dis- 
turbing that the experts the majority seems to rely upon for advice 
in this arena of public health all have extensive ties to the tobacco 
industry. That is the same industry that was found by a federal 
court to have engaged in racketeering and wire fraud in order to 
subvert the public health of American people. And how did they ac- 
complish this fraud? Through a well-documented history of funding 
researchers and third-party groups to cast doubt on the public 
health effects of tobacco. 

Mr. Chairman, this is a serious subject because ultimately this 
is about protecting public health of our citizens. It is about pro- 
tecting the health of our neighbors and our friends and family. If 
the majority is serious about moving forward with this ill-advised 
legislation, then we need to hear from a credible set of witnesses. 
Our citizens deserve no less. I thank you and yield back. 

[The prepared statement of Ms. Johnson follows:] 

Prepared Statement of Full Committee 
Ranking Member Eddie Bernice Johnson 

Thank you Chairman Schweikert. I would like to echo Ms. Bonamici in congratu- 
lating you on being named Chair of the Subcommittee and look forward to working 
with you in this capacity. Unfortunately, I regret that today’s hearing might be a 
rough start in that regard. 
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That is because the “Secret Science Reform Act of 2014” continues one of the most 
regrettable sagas in the history of this esteemed Committee. 

This saga began in the last Congress with Majority requests for data associated 
with studies that the EPA relied upon for certain clean air regulations. It continued 
in August of last year when the Chairman issued the first subpoena from this Com- 
mittee in over 20 years to obtain that same data. And now we are here today, to 
discuss this misguided and mislabeled legislation. 

I want to be clear, the “Secret Science Reform Act of 2014” is built on a false 
premise. None of the science that has been in question during this two year affair 
is “secret.” Is the data protected? Of course it is. 

The data contains tbe personally identifiable health information of hundreds of 
thousands of American citizens. Nonetheless, as the Democratic Minority has re- 
peatedly pointed out, legitimate researchers do have access to this data. 

So what is the problem? What legitimate researchers cannot already access this 
data? At the August 1, 2013, meeting to authorize a subpoena, the Chairman indi- 
cated that Dr. James Enstrom could not access the American Cancer Society data. 
As I have pointed out before. Dr. Enstrom has a long history of conducting research 
and performing consulting work for the tobacco industry. 

And that brings us to today’s hearing. Mr. Chairman, all three of the Majority’s 
witnesses also have significant ties to the tobacco industry. First we have Dr. John 
Graham. While he headed the Harvard Center for Risk Analysis he personally solic- 
ited research funding from Philip Morris. Moreover, he invited Philip Morris public 
relations officials to review a draft chapter of his book on the subject of second-hand 
smoke. Dr. Graham’s Center ultimately received tens of thousands of dollars in 
grants from Philip Morris subsidiary Kraft General Foods. 

Next we have Dr. Tony Cox, who has received numerous research grants from 
Philip Morris tobacco and has collaborated on research with internal Philip Morris 
scientists. In addition. Dr. Cox has served as a litigation consultant for the Philip 
Morris and RJR tobacco companies. 

Finally, we have Dr. Ray Keating. Dr. Keating’s organization, the Small Business 
and Entrepreneurship Council, and its predecessor, the Small Business Survival 
Foundation has solicited and received funding from tobacco companies. Moreover, 
documentation seems to suggest a large amount of collaboration with tobacco com- 
panies. For instance, in the mid-1990’s Dr. Keating released a series of reports on 
FDA tobacco regulations and their negative effects on small business and also filed 
comments with the FDA on the same topic. These reports relied upon a study com- 
missioned by Dr. Keating’s organization and conducted by the American Economics 
Group. 

What Dr. Keating didn’t mention in his reports or FDA comments is that the 
Small Business Survival Foundation was acting as a go-between for the tobacco in- 
dustry. Tobacco company emails show that the study in question was jointly funded 
and organized by Philip Morris and RJR tobacco. 

The reason I highlight this, Mr. Chairman, is that EPA is a public health agency. 
I find it deeply disturbing that the experts the Majority seems to rely upon for ad- 
vice in the arena of public health all have extensive ties to the tobacco industry. 

That’s the same industry that was found by a federal court to have engaged in 
racketeering and wire fraud in order to subvert the public health of the American 
people. 

And how did they accomplish this fraud? Through a well documented history of 
funding researchers and third party groups to cast doubt on the public health effects 
of tobacco. 

Mr. Chairman, this is a serious subject, because ultimately this is about pro- 
tecting the public health of our citizens. It’s about protecting the health of our 
neighbors, and friends, and family. If the Majority is serious about moving forward 
with this ill-advised legislation, then we need to hear from a credible set of wit- 
nesses. Our citizens deserve no less. 

I yield back. 

Chairman SCHWEIKERT. Thank you, Ranking Member Johnson. 

If there are any Members who wish to submit additional opening 
statements, your statements will be added to the record at this 
point. 

Chairman SCHWEIKERT. Our first witness, Hon. John Graham. Is 
it ultimately Professor or Doctor? 

Dr. Graham. Professor. 
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Chairman ScHWEiKERT. Professor Graham, Dean of the School of 
Public and Environmental Affairs at Indiana University. In March 
2001, President George H.W. Bush nominated Dr. Graham to serve 
as Administrator of the Office of Information and Regulatory Af- 
fairs at the Office of Management and Budget. He was confirmed 
by the Senate in July 2001 and served until 2006. Dr. Graham has 
also served as Dean of the Frederick Pardee RAND Graduate 
School, President of the Society of Risk Analysis, Professor of Policy 
and Decision Science at Harvard School of Public Health, and 
Founder and Director of the Harvard Center for Risk Analysis. Dr. 
Graham received his Ph.D. from Carnegie Mellon University. And 
one other just outlier, I think in my graduate school we used one 
of your books. Dr. Graham. 

TESTIMONY OF THE HONORABLE JOHN GRAHAM, DEAN, 
SCHOOL OF PUBLIC AND ENVIRONMENTAL AFFAIRS, 
INDIANA UNIVERSITY 

Dr. Graham. Thank you, Mr. Chairman. You have my written re- 
marks. I just want to use my brief time in the oral session to offer 
a case study of the value of transparency in data access from early 
in my career as an academic. 

In 1981-83 period, I was a doctoral student at Carnegie Mellon, 
as you mentioned. The question I was looking at was do automobile 
safety regulations save lives? The first federal regulations were 
1966 to 1968 in all the cars. They addressed safety belts, padded 
dashboards, collapsible steering columns, and head restraints. They 
all came in at roughly the same time. The engineering estimates, 
based upon laboratory testing, were that these measures would re- 
duce the risk of death in a crash by about 25 to 35 percent. The 
question is would those lives really be saved when they were intro- 
duced in cars in the real world? 

The first real-world valuation was published in 1975 by a pro- 
fessor named Sam Peltzman at the University of Chicago and he 
published it in one of the best peer-reviewed social science journals. 
What Peltzman did was is he assembled national safety data from 
1947 to 1974. He compared the death rates in cars before regula- 
tion and after regulation. His results, which were surprising, were 
that the passenger death rates were down only about seven per- 
cent, not 25 to 35 percent as predicted. And the so-called nonoccu- 
pant deaths — think of pedestrians — were up 20 percent. And as a 
result, the net of it all was he concluded that the regulations didn’t 
save any lives. He then advanced the following theory for why this 
result had obtained. It is now called the theory of risk compensa- 
tion. Drivers, sensing that they are in greater safety, drive faster 
or they are more likely to give their car to their teenage daughters 
or sons thinking they are safe. 

As a young graduate student at Carnegie Mellon, I was quite 
frankly skeptical of this whole study, both the empirical work and 
the theory that was behind it. So I went eagerly and reassembled 
all of Professor Peltzman’s data sources from the documentation 
that he had in his paper. I then reassembled all of his original data 
set since this was all publicly available data. I then re-estimated 
his equations using the equations that were in his article. And I 
found that what he had a purported in his paper was in fact the 



20 


result, given his assumptions. Then I did a reanalysis where I 
added three variables to his equation that he had not considered: 
the growth in the number of small cars in this country, which are 
more lethal than larger cars, the number of heavy trucks on the 
road and the traffic from heavy trucks, and the growth in the num- 
ber of motorcycle registrations because he had included in nonoccu- 
pant deaths not just pedestrians but motorcyclists as well. 

I then reanalyzed the data using his procedure. I found that the 
passenger death rate was about 25 percent lower than would have 
occurred without the regulatory standards and the nonoccupant 
deaths had basically unchanged when you controlled for the growth 
in motorcycling. I concluded that this was a highly successful fed- 
eral regulation that saved thousands of lives. 

With the help of my faculty advisor, we published this reanalysis 
in the peer-reviewed literature. It stimulated a whole bunch of de- 
bate, ten years of additional studies and so forth and so on, and 
I think it is fair to say today that most people would say reading 
this body of evidence that this regulation saved thousands of lives, 
maybe not as many as they originally projected, but a substantial 
number. 

The lessons I would like you to consider from this example, 
which I lived through for years, is that the process of reanalysis 
cannot proceed without transparency of what the data sources are 
and without access to the actual original data to reanalyze the 
problem. Second of all, the reanalysis process is not always 
antiregulation. It is not always antigovernment. In some cases, rea- 
nalysis shows that government regulations work, save lives, reduce 
injuries, and enhance the public good. So the underlying premises 
and assumptions of the bill that we are discussing today in my 
view are politically neutral and they will work for both sides of the 
argument. 

Final comment, when I served in 0MB under President George 
W. Bush, we oftentimes had industry groups and environmental 
groups come to 0MB with their data and analysis of why they 
wanted a regulation changed one way or the other. I think it would 
be a constructive thing if all of that information that they were re- 
quired to give would satisfy these basic standards of transparency 
of what data sources were used and accessibility to the original 
data. That is a neutral — politically neutral outcome that both sides 
of this debate should be subjected to. 

Thank you very much. I look forward to the questions. 

[The prepared statement of Hon. Graham follows:] 
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Topic: Secret Science Reform Act of 2014 

Statement of John D. Graham, Ph.D., Dean, School of Public and Environmental 
Affairs, Indiana University (Bloomington and Indianapolis) and former 
Administrator, Office of Information and Regulatory Affairs, U.S. Office of 
Management and Budget (2001-6). 

Committee on Science, Space and Technology, U.S. House of Representatives 
Tuesday, February 1 1, 2014 


My name is John D. Graham. I am Dean of the School of Public and 
Environmental Affairs (SPEA) at Indiana University (lU), one of the largest public 
affairs schools in the United States and one of the few that combines programs in 
environmental science with programs in public administration. I cannot resist 
reporting that, in the most recent rankings published by U.S. News and World 
Report . lU-SPEA was ranked #2 out of 266 programs with an accredited Master’s 
Degree in Public Affairs (MPA). In this survey, for the first time in history, lU- 
SPEA was ranked ahead of Harvard’s Kennedy School of Government and is the 
highest ranked MPA program at a public university in the United States. 

Prior to serving at lU, I served as Dean of the Pardee RAND Graduate School at 
the RAND Corporation in Santa Monica (2006-8), Administrator of the Office of 
Information and Regulatory Affairs (OIRA) at the U.S. Office of Management and 
Budget (0MB), and Professor of Policy and Decision Sciences at the Harvard 
School of Public Health, where I founded and led the Harvard Center for Risk 
Analysis (1985-2001). 

With regard to my educational background, I earned a BA in politics and 
economics at Wake Forest University (1978), an MA in public affairs at Duke 
University (1980), and a Ph.D. in urban and public affairs at Camegie-Mellon 
University (1983). Before joining the Harvard faculty in 1985, 1 completed a post- 
doctoral fellowship in environmental science and public policy at the Harvard 
School of Public Health. 

I am pleased that the Committee is giving priority to the topics of transparency, 
reproducibility, and quality in the science at the U.S. Environmental Protection 
Agency (EPA). This topic has been a strong interest of mine for decades. In 1988 
I co-authored In Search of Safety: Chemicals and Cancer Risk (Harvard University 
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Press), which examined the scientific and political aspects of the federal 
government’s regulation of two industrial chemicals: formaldehyde and 
benzene. In particular, we reviewed the relevant risk assessments at EPA, the 
Occupational Safety and Health Administration (OSHA), and the Consumer 
Product Safety Commission (CPSC). In 1991 I edited Harnessing Science for 
Environmental Regulation (Praeger Press), which explored alternative institutional 
models for strengthening the quality of science at EPA, including the constructive 
contributions of EPA’s Science Advisory Board. In the late 1990s, I also worked 
with teams of environmental scientists from multiple countries to develop norms in 
the quest for greater transparency and reproducibility in environmental science 
studies that support regulatory decision making. Most recently, I organized an 
international team of scientists and practitioners to examine the scientific and 
policy aspects of regulating persistent, bioaccumulative and toxic chemicals 
http://www.indiana.edu/~spea/facultv/pdf/scientific policy analysis of persistent 
bioaccumulative and toxic chemicals PBT .pdf . 


STATEMENT OF THE PROBLEM 

No scientific organization can produce data, analytic results, and interpretations 
that are completely free of bias or error or misleading 
interpretation. Organizations, whether they be universities, think tanks, 
government laboratories or regulatory bodies are imperfect. They are imperfect in 
part because they are staffed by human beings, and people are 
imperfect. Consequently, mechanisms have been developed to enhance the quality 
of science produced by organizations. Those mechanisms range from strategies to 
enhance the scientific training of the personnel who perform the work to 
procedures of internal and external peer review to ensure that scientific products 
meet applicable information-quality objectives. 

The quest for greater quality in the scientific products at EPA is particularly 
challenging for several reasons. 

First, EPA’s standards of quality generally need to be quite high, since the reports 
that are issued have an important impact on regulation, public health, 
environmental quality, affected businesses and workers, and the economy as a 
whole. An analytic mistake in an EPA report can cause numerous people to 
become sick or lose their lives due to inappropriately high levels of environmental 
pollution. If erroneous information about the hazards of an industrial chemical or 
pesticide is disseminated, the affected company can lose an entire product line and 
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people can lose their jobs. Entire sectors of the U.S. economy (e.g., energy, 
manufacturing, and agriculture) are strongly impacted by EPA regulation, and thus 
it is crucial that EPA’s scientific determinations achieve a high level of quality. 

Second, the scientific culture at EPA is fragile and still at an early stage of 
development. In my experience working with EPA, I have found that the political, 
legal and engineering cultures are fairly strong but the cultures of science and 
economics are highly variable across the agency’s programs. The uneven role of 
science at EPA should not be surprising because EPA is seen as a mission-oriented 
agency more than as a science agency, and first-rate scientists who are interested in 
public service employment might be more inclined to launch a career at the 
National Institutes of Health or the National Science Foundation or the National 
Research Council/National Academy of Sciences than a career as a scientist at 
EPA. For years EPA has been tciking constructive steps to enhance the scientific 
culture of the agency and there are numerous outstanding scientists working at the 
agency. Nonetheless, the effort to build a culture of sound science at EPA is a 
work in progress. 

Third, the headquarters office for EPA, where many of the key regulatory 
decisions are made, is located in Washington, DC but many of the agency’s top 
scientists are located elsewhere (Research Triangle Park, North Carolina; 
Cincinnati, Ohio; and Ann Arbor, Michigan, for example). From the sheer 
perspective of physical location, many scientists at EPA are more at the periphery 
than at the center of the Agency’s decision making. 

Fourth, the field of environmental science is full of uncertainties and the literature 
is constantly exploding with new scientific results and alternative interpretations of 
previous results. Keeping abreast of this field is quite challenging, as 
environmental science has to be one of the most dynamic fields within the physical 
and life sciences. Staying up to date is particular challenging because of the 
multiple disciplines that contribute information, everything from environmental 
epidemiology and toxicology to nanotechnology and atmospheric chemistry. 

Finally, the credibility of one of the Agency’s most important scientific tools, risk 
assessment, is constantly under attack. Industry says the agency’s risk assessments 
rely too much on conservative, default eissumptions. Environmental groups say the 
agency’s risk assessments are preoccupied with cancer and are downplaying the 
importance of persistence, bioaccumulation, and endocrine disruption in setting 
priorities for chemical risk assessment and regulation. 
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A series of reports from the National Research Council/National Academy of 
Sciences over the last fifteen years has documented persistent shortcomings in the 
quality, transparency and reproducibility of the agency’s scientific 
determinations. Those reports have addressed specific substances such as fine 
particulate matter, dioxin, TCE, formaldehyde, to name just a few. A more recent 
report, Science and Decisions: Advancing Risk Assessment (NRC, 2009), 
addresses the way science and risk assessments are used in support of decision 
making at the agency. 

TWO CASE STUDIES FROM MY OMB EXPERIENCE 


During my tenure as OIRA administrator, I was periodically drawn into issues 
where EPA science issues consumed my time in ways that I could not have 
predicted. Here are two examples where shortcomings in EPA science caused 
significant diversion of energy inside the Executive Office of the President. 

Example #1: perchlorate contamination 

In the 2002-3 period, my boss at OMB, Mitch Daniels, called me into his office 
and asked me what I was doing about “perchlorate”. I confessed that I did not 
know what he was talking about; indeed I thought he was referring to the solvent 
perchloroethylene that is widely used in diy cleaners. He was not. He was talking 
about a substance (primarily ammonium perchlorate) that is used as an oxidizer in 
solid rocket fuels and propellants for munitions. When rockets and other 
munitions are tested at US military sites, it is not uncommon for residual 
perchlorate to end up in the soil and in nearby surface water. 

EPA was concerned about the perchlorate contamination. On the other hand, at 
that time, our troops in Afghanistan and Iraq were stretched to the limit, even 
though defense spending was growing rapidly. OMB could not duck a key 
resource-allocation issue: Would it be wise to spend billions of dollars cleaning up 
small concentrations of the perchlorate at military installations or to dredge the 
Colorado River in a quest to clean up sediments that contained perchlorate? I 
agreed to look into the issue. 

I went back to my office and requested, along with the White House Office of 
Science Technology, an EPA briefing on perchlorate. EPA’s response was 
straightforward. In 1985 perchlorate contamination had been detected in drinking 
water wells near California Superfund sites but it was not until 1997 that EPA 
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detected a national contamination issue. Apparently, more than 1 1 million people 
in the United States have perchlorate in their drinking water above the agency’s 
minimum reporting level of 4 parts per billion (4 micrograms per liter). The 
reporting level is not a safety threshold but simply the lowest level that is required 
to be reported to the agency in its data system. No national drinking water 
standard for perchlorate had been set, the type of EPA standard that would be 
designed to protect public health. EPA had issued in 2002 a draft risk 
characterization for perchlorate that implied that a drinking water standard of 1 
part per billion would be necessary to protect public health with an appropriate 
margin of safety. This figure is a factor of 30 more stringent than the interim 
guidance on perchlorate that EPA had issued in 1998 during the Clinton 
administration. EPA was working through comments on the new draft risk 
characterization and had already received scientific advice from a committee of the 
EPA’s Science Advisory Board. 

In a second meeting at OSTP, I learned that a bitter dispute about the safe level of 
exposure to perchlorate was being waged by scientists at EPA, the National 
Aeronautics cind Space Administration (NASA), the Department of Defense 
(DOD) cind the Department of Energy (DOE). EPA was relying on a standard 
analysis of animal data and protective uncertainty factors to make its 
determination; NASA, DOD and DOE were arguing that available human data 
could support a much more permissive safety determination, though their analytic 
approach would be different than the one EPA normally uses. The issue was 
important because the most sensitive group for exposure to perchlorate were the 
fetuses of pregnant women who might have hypothyroidism or iodide deficiency. 

OSTP and OMB decided, after multiple meetings with the agencies, that it was 
best to refer the issue to an appropriate committee of health scientists at the 
National Research Council of the National Academy of Sciences. The relevant 
agencies reluctantly agreed, recognizing that the NRC/NAS review would take 
additional time. All of the agencies contributed funds to make the NAS report 
possible, not knowing what NRC/NAS would conclude. 

To make a long story short, the NRC/NAS committee, after hearing the views of 
all the agencies and reviewing the scientific literature, concluded in 2005 that the 
EPA analysis of animal data was inappropriate and the human data on exposures to 
perchlorate should be utilized to determine a safe level of exposure. The 
committee went further and produced an analysis suggesting that EPA’s draft 
figure of 1 part per billion was a factor of 24 too low, meaning that public health 
would not be at risk, even at much higher concentrations of perchlorate in drinking 
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water. (See National Research Council. Health Implications of Perchlorate 
Ingestion . National Academy Press, 2005.) I have not followed this issue closely 
since I left the federal government in 2006 but my understanding is that EPA has 
still not proposed a national drinking water standard for perchlorate, though they 
have plans to do so (see 

http://water.epa.gov/drink/contaminants/unregulated/t)erchlorate/cfm ). 

Example #2: fine particle pollution 

According to EPA and OMB reports, one of the most beneficial suite of 
regulations issued by the federal government is a set of air quality rules aimed at 
reducing human exposure to fine particulate matter (sometimes called 
soot). Particles vary not only in size but in chemical composition. Carbon- 
containing particles are emitted directly by diesel engines while sulfate and nitrate 
particles may be formed in the atmosphere after the gases sulfur dioxide and 
nitrogen dioxide are emitted from electric power plants or other sources. From my 
faculty colleagues at the Harvard School of Public Health, I was already aware of 
two curious features of EPA’s scientific position on particles. 

First, EPA was assuming that all particles are equally toxic, regardless of their 
chemical composition. (By way of contrast, scientists at the European 
Commission in Brussels were producing benefit estimates in 2005 suggesting that 
sulfates are much less toxic than other forms of particles). Second, EPA was not 
reporting much uncertainty in its estimates of the benefits of reducing human 
exposures to fine particle matter. Thus, when EPA forecasted that (say) 1,000 
lives might be saved from a particular regulation, it was not clear whether the truth 
could fall between 900 and 1 100 or whether the truth could be anywhere from 0 to 
10,000. I devoted three years of effort with my staff at OMB, nudging EPA in the 
direction of addressing these two curious features of their scientific position on the 
health benefits of reducing particle pollution. 

I made more progress on the second issue than the first issue, in part because in 
2002 the National Research Council/National Academy of Sciences released a 
report recommending that EPA do a better job of quantifying the degree of 
uncertainty in the agency’s estimates of public health benefit from reduced 
exposure to particles. (See National Research Council. Estimating the Public 
Health Benefits of Proposed Air Pollution Regulations . National Academy Press, 
2002.) By 2005 EPA was reporting quantitative uncertainty estimates that 
addressed some of the fundamental uncertainty about the pollution-mortality 
relationship. 
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My takeaway message from EPA’s uncertainty work was not that EPA’s lifesaving 
estimates were biased on the high side but that the truth could easily be a factor of 
3 lower or higher than the agency’s primary estimates. When all of the 
uncertainties in EPA’s benefit analysis for particles are considered, I became 
convinced that decision makers should recognize that uncertainties were quite 
large indeed. 

In the 2001-2006 period, one of the key barriers to doing comprehensive 
uncertainty analysis of particle-benefit estimates was that the original health data 
from the underlying epidemiology studies (e.g., the Harvard Six-Cities Study and 
the American Cancer Society cohort) were not being made publicly available to 
researchers through EPA’s web site. I checked briefly on EPA’s web site prior to 
preparing this testimony and - 10 years later - EPA has still not made the 
underlying data publicly available on their web site. To their credit, EPA did - in 
the 2000-2003 period subject the key particle-oriented health studies to a peer 
review, replication, and re-analysis by an independent team of scientists at the 
Health Effects Institute. Although the HEI replication did not discover any major 
errors, HEI did find some instability in coefficients depending on which pollutants 
are included in the statistical models (e.g., some of the results for sulphur dioxide 
and sulfates were unexpected). If the underlying data from the key health studies 
were made publicly available for all researchers to analyze (rather than just a select 
few appointed by EEI), I think it is quite possible that many new insights would be 
gleaned and some of the conventional wisdoms we now accept as fact would be 
dislodged or refined. 

I made little progress at 0MB trying to nudge the agency toward a more plausible 
position on the relative toxicity of different types of particles. The agency holds to 
a default position that all fine particles are equally toxic, and does not do any 
quantitative uncertainty analysis of this assumption. In fairness to the Agency, 
they have been supporting more than a decade of research at the Health Effects 
Institute (Cambridge, MA) on the relative effects of different particles but I am not 
aware that any of this HEI-sponsored research has yet had any quantitative impact 
on EPA estimates of the benefits of fine particle control. 

I trust that these two case studies, from the trenches of my OMB experience, 
illustrate some of the challenges in bringing more rigor and humility to the 
scientific determinations of EPA. 

OMB EFFORTS ON DATA ACCESS AND REPRODUCIBILITY 
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In October 1 999 0MB issued Circular A-1 10 on data access that was intended to 
spur agencies to make available the original data from federally-funded 
studies. 0MB actually softened what was intended by the underlying legislation 
(the “Shelby Amendment”) because OMB applied the data access requirement 
only to new studies, not to those that were initiated before the legislation and 
before OMB’s circular were finalized. As a result, OMB’s data access policies are 
not having much impact on public access to the key health data that support EPA’s 
air quality regulations. 

When I served at OMB from 200 1 to 2006, 1 led a government-wide effort to 
establish information-quality (IQ) guidelines pursuant to new information-quality 
legislation that was passed by Congress before the Bush administration began. We 
established IQ guidelines at OMB and then I supervised a process whereby each 
agency, including EPA, established their own IQ guidelines. A central theme of 
these IQ guidelines was transparency and reproducibility of original data and 
results from analytic models. The key operational feature of the agency guidelines 
was an opportunity for the public to petition an agency and accomplish a correction 
of agency information that does not meet the agency’s information-quality 
guidelines. Unfortunately, the impact of those guidelines has been weakened 
because federal courts have ruled that the agency’s IQ guidelines are not legally 
binding. Outside parties (e.g., businesses, labor unions and environmental groups) 
do not have access to the courts to seek relief in situations where a petition is 
handled arbitrarily by an agency, and thus the agencies know that the law has little 
teeth. 

CQMMENTS QN THE “SECRET SCIENCE REFORM ACT OF 2014” 

The legislation you have asked me to comment upon is commendably short, and it 
is a proposed amendment to Section (b) of the Environmental Research, 
Development, and Demonstration Authorization Act of 1978. In summary, EPA is 
not permitted to issue regulations (or other “covered actions”) unless (A) all 
scientific and technical Information relied upon is specifically identified; (B) such 
information is publicly available in a manner that is sufficient for independent 
analysis and substantial reproduction of research results. 

I fully support point A, as it is an elementary principle of transparency. A third 
party (or even another federal agency or OMB) cannot possibly evaluate the merits 
of a covered action if they do not know what specific scientific and technical 
information was relied upon by EPA. 
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I also support point B, which I believe is the heart of the legislation, as it requires 
both public access to the scientific and technical information, and access in a form 
that facilitates independent analysis and reproduction of research results. With 
regard to agency compliance, what I envision is simply a link on EPA’s web site — 
one for each covered action ~ that contains one or more files of original scientific 
and technical information (including original data and analytic models and 
guidance about how to access and utilize the files) that are sufficiently detailed that 
a third party could process the information and thereby substantially reproduce the 
results that the agency is relying upon. The word “substantially” is preferred to a 
word such as “completely” because a reproduction within a certain number of 
significant figures is certainly adequate in the vast majority of cases. Or there may 
be cases where the agency results - or the results of a third party — contain a 
harmless numerical error that is judged to satisty the standard of substantial 
reproduction but would not satisfy the standard of complete reproduction. 

From a practical point of view, agency compliance with the public access provision 
is most straightforward for studies that the agency funds in the future. The agency 
will simply require the grantee to include with their final report files of data and 
guidance that satisfy the provisions of this legislation. The National Science 
Foundation already has a procedure for grantees to submit original information to 
NSF at the end of a grant period, and EPA could look at the NSF procedure as a 
possible model. NIH may also have some useful guidance of this sort. For a 
previously-funded EPA study (i.e., prior to enactment of this legislation), 
compliance should be addressed only at the point that EPA staff determine that 
they intend to use such a study in support of a covered action. At that point, 
agency staff should reach out to the relevant author(s) and seek submission of the 
underlying information. There may be situations where the authors no longer have 
possession of the underlying information (e.g., in the case of older studies) or do 
not have the time or money to preptu'e the research results in the form that EPA 
requests. In such cases, EPA may need to establish a consulting or contractual 
arrangement to obtain the underlying information or may decide instead to rely on 
a different source of scientific and technical information for the covered action. 

Since the agency often relies on scientific and technical information that the 
agency has not funded (e.g., university-funded research in the peer-reviewed 
literature, technical submissions by scientists from industry and environmental 
groups, information submitted by other federal agencies, and so forth), compliance 
with this legislation may be facilitated by an EPA rule or guidance that explains 
how public access to such information will be accomplished. In the course of 
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preparing covered aetions, ageney personnel will often need to reaeh out to the 
authors of seientifie and teehnieal information and request that the underlying 
information that the Ageney intends to rely upon be submitted to the Ageney in a 
partieular form. This type of information request and exehange is common 
throughout the scientific community on a day-to-day basis. In fact, many authors 
of scientific papers are now posting on their web sites supplementary information 
that supports a paper that has been published in a peer-reviewed journal. The 
supplementary information may include original data and descriptions of analytic 
models and computer code. In some cases, identifiers are removed to protect the 
confidentiality of human subjects. There may be some segments of the scientific 
community that perceive new data-access requirements to be onerous (particularly 
as they relate to older studies) but the vast majority of the environmental scientific 
community should not have difficulty satisfying the public access provision in the 
draft legislation. The last sentence does not mean that the Committee will not hear 
complaints from scientific societies about this legislation. I predict that you will 
hear complaints about how practical issues will be worked out but it is important to 
remember that the scientific community is not accustomed to being told by the 
government how to assemble and disclose their work products, particularly 
products that were published years ago. Working out the kinks in this process will 
take some time and consensus building. 

I have already alerted Committee staff that the Institute of Medicine (Washington, 
DC) will soon by holding a public workshop (March 19, 2014) to address how 
“data sharing” should be accomplished in the field of environmental health, 
especially when data from human subjects are at issue. The summary of the charge 
to the workshop indicates that “Environmental health experts agree that the 
question is not “if’ research data should be shared but “how.”” I encourage 
Committee staff to attend the workshop (Workshop on Principles and Best 
Practices for Sharing Data from Environmental Health Research, Institute of 
Medicine, March 19, 2014, 8:30 AM, 

http://www.iom.edu/activities/environment/environmentalHealthRT/2014-Mar- 

19.asp x). 

With regard to the “covered actions,” it is extremely reassuring that this language 
goes beyond “regulations” to Include technical documents used to support federal 
regulations (e.g., regulatory impact analyses) and scientific and technical 
documents issued by EPA that are known to influence subsequent regulations by 
EPA, the states and international bodies. In my experience, some of the most 
significant actions taken by EPA are scientific determinations rather than 
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regulations per se and they are often issued as guidance (e.g., IRIS determinations 
about chemicals are normally considered guidance). 

With regard to (2) the clarification and exclusion regarding disclosures that are 
prohibited by law, I assume this covers confidential business information, some 
types of security-related information (e.g., how a chemical is used by the Defense 
Department in a particular weapon system) and possibly some privacy laws. I am 
not aware of any other kinds of disclosures that would be prohibited by law. 

SUPPORT FROM ORGANIZATIONS 

In conclusion, I would like to cite support for data access and reproducibility that 
was previously stated by authoritative organizations. 

1 . Administrative Conference of the United States (excerpts below) 
http://acus.gov/sites/default/files/documents/Science%20Recommendation%20AP 
PROVED-FINAL l.pdf 

“Disclosing Underlying Studies and Data. To the extent practicable and permitted 
by law and applicable policies, each agency should identify and make publicly 
available (on the agency website or some other widely available forum) references 
to the scientific literature, underlying data, models, and research results that it 
considered. In so doing, the agency should list all information upon which it relied 
in reaching its conclusions, as well as any information material to the scientific 
analysis that it considered but upon which it ultimately did not rely. Consistent 
with the limitations in the Information Quality Act (IQA) guidelines issued by the 
Office of Management and Budget and its own IQA guidelines, each agency 
should ensure that members of the public have access to the information necessary 
to reproduce or assess the agency’s technical or scientific conclusions.” 

“Data Disclosure. To the extent practicable and in compliance with applicable 
legal restrictions, privileges, protections, and authorities, agencies should seek to 
provide disclosure of data underlying scientific research, including both privately 
and federally funded research being considered by the agencies. Where practicable, 
such information should be disclosed in machine-readable format. Where such data 
are not subject to legal or other protections, and the data’s owners nonetheless will 
not provide such access, agencies should note that fact and explain why they used 
the results if they chose to do so. Agencies should review their confidential 
business information policies to ensure that they include appropriate mechanisms 
to prevent over-claiming.” 
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2. The submission guidelines for the prestigious journal Science . 

“As a condition of publication, authors must agree to make available all data 
necessary to understand and assess the conclusion of the manuscript to any reader 
of Science” 


Thank you in advance for the invitation to testify. Please do not hesitate to contact 
me if I can answer any questions or if I can supply any additional information. 
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Biographical Sketch 
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executive. He earned his B.A. (politics and economics) at Wake Forest University (1978) where he won 
national awards as an intercollegiate debater. He earned his M.A. degree in public policy at Duke University 
(1980) before serving as staff associate to Chairman Howard Raiffa’s Committee on Risk and Decision Making 
of the National Research Council/National Academy of Sciences. His Camegie-Mellon University Ph.D. 
dissertation on automobile safety, written at the Brookings Institution, was cited in pro-airbag decisions by the 
U.S. Supreme Court (1983) and by Secretary of Transportation Elizabeth Dole (1985). 

Dr. Graham joined the Harvard School of Public Health as a post-doctoral fellow in 1 983 and as an assistant 
professor in 1985. He taught the methods of decision analysis and cost-benefit analysis to physicians and 
graduate students in public health. His prolific writings addressed both the analytic and institutional aspects of 
lifesaving policies. Dr. Graham earned tenure at Harvard in 1991 at the age of thirty-four. 

From 1990 to 2001 Dr. Graham founded and led the Harvard Center for Risk Analysis (HCRA). By raising 
over $10 million in project grants and philanthropic contributions. Dr. Graham helped support eight new faculty 
positions and dozens of post-doctoral and doctoral students. By 2001 HCRA became internationally recognized 
for analytic contributions to environmental protection, injury prevention, and medical technology innovation. 

In 1 995 Dr. Graham was elected President of the Society for Risk Analysis (SRA), an international membership 
organization of 2,400 scientists and engineers. Dr. Graham reached out to risk analysts in Europe, China, Japan 
and Australia as he helped organize the first World Congress on Risk Analysis (Brussels, 2000). Later, in 2009, 
Dr. Graliam received the SRA’s Distinguished Lifetime Achievement Award, the society’s highest award for 
excellence. 

In March 2001 President George W. Bush nominated Dr. Graham to serve as Administrator, Office of 
Information and Regulatory Affairs, Office of Management and Budget. He was confirmed by the Senate in 
July 2001. Located in the Executive Office of the President, this small office of 50 career policy analysts 
oversees the regulatory, information and statistical activities of the federal government. In this capacity, Dr. 
Graham worked to slash the growth of regulatory costs by 70% while encouraging good regulations that save 
lives, prevent disease, and protect the environment. 

From March 2006 to July 2008 Dr. Graham was Dean of the Frederick Pardee RAND Graduate School at the 
RAND Corporation in Santa Monica, California. PROS is the largest doctoral program in policy analysis in the 
world. In this role, Dr. Graham streamlined the core curriculum, established new analytic concentrations, 
revised the program requirements to enable students to launch their dissertations more promptly, and raised 
funds from individuals and corporations to support scholarships, dissertation support and policy papers co- 
authored by students and RAND researchers. 

On July 28, 2008 Dr. Graham assumed the Deanship of the Indiana University School of Public and 
Environmental Affairs (Bloomington and Indianapolis), one of the largest public policy schools in the United 
States. The School has about 1,500 undergraduate majors, over 300 master’s students and about 80 doctoral 
students. The 75 full-time faculty include laboratory scientists, social scientists, lawyers and policy specialists. 
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Chairman SCHWEIKERT. Thank you, Professor Graham. 

Our next witness is Dr. Tony Cox, Chief Sciences Officer at Next 
Health Technologies, Clinical Professor of Biostatistics and 
Informatics at Colorado Health Sciences Center, and President of 
Cox Associates. Next Health Technologies offers advanced data 
analytics solutions to healthcare plans to reduce health, financial, 
and member attrition risks. Dr. Cox is also the current editor-in- 
chief of the journal Risk Analysis. In 2012 he was inducted into the 
National Academy of Engineering and is a member of the National 
Academies Standing Committee on the use of public health data. 
Dr. Cox received his Ph.D. in risk analysis from MIT. Dr. Cox. 

TESTIMONY OF DR. LOUIS ANTHONY COX, JR., 

CHIEF SCIENCES OFFICER, NEXT HEALTH TECHNOLOGIES, 

CLINICAL PROFESSOR, BIOSTATISTICS AND INFORMATICS, 
COLORADO HEALTH SCIENCES CENTER, 

AND PRESIDENT, COX ASSOCIATES 

Dr. Cox. Thank you for inviting me today to discuss whether the 
data underpinning regulations should be made openly available. I 
am testifying on my own behalf today in support of the Secret 
Science Reform Act. I have provided the Committee the detailed 
CV describing my academic, publishing, and business affiliations. 

I am a risk analyst and I am happy to tell you why I think ac- 
cess to data is essential for high-quality analysis in the public in- 
terest. I can also tell you that it is not easy to get such access. Ms. 
Johnson or others with similar views might decide that researchers 
like me who have worked with cigarette manufacturers to quantify 
risks of smoking-associated diseases are not legitimate enough to 
deserve access to data, but without such access, we cannot correctly 
quantify what the risks are. 

We are discussing a key question for science and policy today. Is 
the public interest best served by requiring that data behind 
science-based environmental regulations be made available to those 
who want to see it? Many who argue yes believe that the very es- 
sence of trustworthy science is reproducibility of results and shar- 
ing of the data said to drive them. For example, over 2/3 of recently 
surveyed professionals involved in risk assessment said it was very 
important to have access to the underlying raw data so that they 
could independently analyze the results, but only about 1/3 said 
that such access was usually the case. The proposed Secret Science 
Reform Act would help to close this gap. 

A concern about sharing of data is that it might prove burden- 
some for the original investigators, exerting a chilling effect on 
their research, but keeping well-organized records, data, and lab 
notebooks so that others can check methods and results is or 
should be part of the training of every good scientist. It imposes no 
extraordinary burdens and has many benefits. Scientific journals 
can also facilitate sharing of the data behind published conclusions. 

A second concern expressed by ALA and others is that making 
study data available might threaten the privacy of individuals. We 
have already heard that this morning. The technical issue of how 
to protect privacy while allowing valid statistical analysis is best 
addressed by technical solutions, and many excellent one such as 
multiple imputation are now available. They are already being 
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used successfully at the Census Bureau and elsewhere. So I think 
this concern is a bit of a red herring. We can meet it by applying 
existing technical methods. 

But the most important concern I suspect is not technical. It is 
that bad people or people with agendas other than pure science and 
the public interest might delay good regulations by performing 
untrustworthy new analyses and reanalyses that would obscure the 
need for action. To address this concern, I think we must candidly 
assess how well our current scientific process delivers trustworthy 
results without much pressure from external reanalyses of data. It 
does not. 

We are living in an age of catastrophic failure in the reproduc- 
ibility and trustworthiness of scientific results as evidenced by arti- 
cles such as “Why Most Published Research Articles Are False” 
from 2005 and “Trial and Error: Why Science Is Failing Us” from 
2011 or an editorial just last month on reproducibility in Science 
magazine. A common theme is that there is too much pressure on 
original investigators to use dubious statistical methods to publish 
results that are sensational but not necessarily correct and there 
is not enough encouragement for original investigators to do unbi- 
ased research knowing that others will soon be reanalyzing their 
data and claims. Fixing this critical problem requires more scrutiny 
and greater access to original data, not less. 

Let me end with two examples from my own experience in public 
health risk analysis. First, by applying causal analysis methods to 
the publicly available national mortality and morbidity air pollu- 
tion study data, I recently discovered that air pollution levels are 
indeed correlated with mortality risks in 100 U.S. cities. This was 
already well known. For example, both were associated with cold 
winter days. But surprisingly, there was no evidence that reducing 
air pollution has caused any reductions in mortality rates. Open ac- 
cess to the data makes such unexpected discoveries possible and 
encourages others to check and possibly improve upon the results 
potentially informing important public policy. 

As a last example, Dublin, Ireland, recently extended bans on 
coal burning based on research claiming that banning coal burning 
immediately reduced mortality rates. That research was done and 
publicized in part by U.S. investigators who have prominently 
shaped U.S. EPA’s science and claims about air pollution health ef- 
fects. Yet a reexamination of the data last year funded by the 
Health Effects Institute revealed that its major conclusion was not 
true; mortality rates did not come down any faster where coal 
burning was banned from where it wasn’t. European researchers 
had already pointed out years ago the fallacy of assuming that just 
because pollution levels in mortality rates had both declined, that 
suggested that one caused the other. But without access to the 
original data, they cannot quickly and easily prove that the original 
conclusions did not follow from the data. By the time the original 
U.S. investigators were funded to take another look at the data, 
Irish public policy had already been made. Only ready access to the 
data would have enabled others to fix the problem in time to in- 
form policy decisions. 

We need not repeat such experiences here. We can choose to 
make data used to support regulatory decisions openly available for 



36 


others to analyze and not wait until policy has been made and 
changes enacted before allowing the public to find out whether bet- 
ter analyses would have led to different results. I believe that doing 
so will promote sounder science and hence strongly promote the 
public interest. 

Thank you for your attention. 

[The prepared statement of Dr. Cox follows:] 
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Chairman Schweikert and Members of the Subcommittee, thank you for inviting me to discuss 
whether regulatory science and the data on which it rests should be made openly available. I am 
testifying on my own behalf today, in support of the proposed Secret Science Reform Act. I need access 
to data for my work on health risk assessment, and am grateful for this opportunity to explain why. I have 
provided the Committee members with a detailed CV describing my academic, publishing, and consulting 
affiiiations- 

We are discussing a question of great current public, policy, and scientific interest: Is the public 
interest well served by requiring that data used to support policy decisions be made available to those 
who want to see it? M^y who argue yes believe that the very essence of good science is reproducibility 
of results, and sharing of the observations and data that are said to drive them (Cox, 2009, p. 5). For these 
people, openness to scrutiny is a hallmark of sound scientific reasoning, and a prerequisite for sound 
scientific process and fortrustworthy conclusions. Many scientists and analysts themselves are of this 
persuasion. For example, a recent survey of three professional societies involved in risk assessment 
found that “69 percent said it was ‘very important’ to have access to the underlying raw data for the most 
critical studies in order to do their own independent analysis of the results.” However, “only 36 percent 
said that having this access was often or always the case” (Butterworth, 2013). The proposed Secret 
Science Reform Act will help to address that gap. 

Those who oppose requiring open sharing of data used to support regulations and policies 
typically cite several concerns (e.g., Neutra et a1., 2006; Pearce and Smith, 2011). One is that the process 
might be abused by unscrupulous parties. Like the tobacco industry, others might seek to “manufacture 
doubt” to obscure the clear implications of good science and to delay socially beneficial actions by 
proposing alternative, inferior analyses, A second concern is that divulging data might threaten the 
privacy of individuals included in study populations. A third concern is that requiring data to be shared 
might prove burdensome for the original investigators, exerting a chilling effect on research in the public 


interest. 
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To these three objections, taken in reverse order, it might be replied that, first, the habit of 
keeping well-organized and documented records, data, and lab notebooks in expectations that others will 
use them later to try to independently reproduce and verify important claimed findings is ~ or should be - 
part of the training of every good scientist. No extraordinary burden is imposed by such good practices, 
Transparency of data and methods and scrutiny of results by others, perhaps using different methods, is 
something that scientists should expect and welcome. There is also much that scientific journals can do, 
and are doing, to encourage data transparency and to facilitate making documentation of data, models, 
and analyses readily available to those who want to use them. 

Second, the concern that making study data available could threaten the privacy of individuals 
rests on an important, but purely technical, statistical issue: Do statistical data in fact allow individual 
attributes or facts that should be protected to be discovered? This technical problem is best addressed by 
technical solutions, and many excellent ones are now available to allow statisticians to do valid analyses 
while protecting individual data (Reiter, 2009; Klein et al., 2013). These methods, such as multiple 
imputation, have already been extensively developed, tested, and successfully applied, at the Census 
Bureau and elsewhere. So, I think that this concern should be viewed as a bit of a red herring: appropriate 
technical methods to handle it are already available and are being used in other areas. 

But the most important concern, I suspect, is often not technical. It is about human behavior, and 
incentives, and the sociology of science. This is the concern that bad people will delay good regulations 
and remedial actions by misusing data and performing untrustworthy analyses to mislead the public and 
policy makers (Neutra et al., 2006). Such concerns have long been expressed about the use of risk 
analysis and technical analysis more generally (Silbergeld, 1993). To address them, 1 think we must 
candidly assess how well the scientific process delivers trustworthy results without much pressure from 
independent examination and reanalysis of data. It does not. We are now living in an age of catastrophic 
failure in the reproducibility and trustworthiness of scientific results, as witnessed by articles such as 
“Why most published research articles are false” (loannidis, 2005), “Trials and errors: Why science is 
failing us” (Lehrer, 201 1), and “Beware the creeping cracks of bias” (Sarewitz, 2012). In the January 17* 
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issue of Science magazine this year, Editor-in-^l^hief Marcia McNutt noted that a worrisome proportion of 
peer-reviewed published results are not reproducible, and she armounced plans to expand their editorial 
board, with advice from the American Statistical Association, “to ensure that manuscripts receive 
appropriate scrutiny in their methods of data analysis.” A common theme is that there is too much 
pressure on original investigators to use dubious statistical methods to publish results that are sensational 
but not necessarily true (false positives), and there is not enough encouragement for investigators to do 
high-quality, reproducible research, with the confident expectation that others will soon be looking over 
their shoulders and reanalyzing their data, perhaps using less biased methods. To fix what is manifestly 
broken takes more scrutiny and greater access to data, not less. As for the very legitimate fear that those 
who disagree with us might use open access to data and reanalyses to confuse and delay actions that we 
favor, this has been part of the cost and a great part of the benefit of free, democractic societies since well 
before John Stuart Mill wrote, in On Liberty, that “Wrong opinions and practices gradually yield to fact 
and argument: but facts and arguments, to produce any effect on the mind, must be brought before it. ... 
The beliefs which we have most warrant for, have no safeguard to rest on, but a standing invitation to the 
whole world to prove them unfounded.” The best defense against unscrupulous use or motivated 
interpretations of data - whether from regulators or from industry or from anyone else - is to make it 
openly available, so that the grounds of debate turn from who is privileged to see the facts to how one 
should best interpret them. 

Let me end with two recent examples from my own experience in public health risk analysis. 

First, the public availability of the National Mortality and Morbidity Air Pollution Study (NMMAPS) 
data set recently allowed me to apply econometric tests for potential causality to air pollution and 
mortality data from 1 00 U.S. cities. An unexpected finding was that, although levels of air pollution are 
significantly associated with levels of elderly mortality rates (and both are associated with cold winter 
days), there is no evidence that reductions in air pollution levels have caused any reductions in mortality 
rates (Cox, 2012). This was a new finding from old data, using methods that other investigators had not 
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tried. It may be important information for policy-makers to consider. I hope that others will repeat and 
improve upon my analysis. Without open access to the data, that would not be possible. 

Second, in 2012, Dublin extended bans on coal burning (DECLG, 2012) because of research 
(Clancy et al., 2002), done in part by U.S. investigators who have prominently shaped U.S. EPA beliefs, 
assuring them that cutting coal-burning had promptly and obviously reduced mortality rates, especially 
cardiovascular deaths (Harvard School of Public Health, 2002). A closer look at the data in 2013, funded 
by the Health Effects Institute, revealed that this was not true: these mortality rates did not decrease any 
faster where coal burning was banned than where it wasn’t (HEI, 2013). The original investigators had 
not accounted for the genera! trend that mortality rates were coming down all over Ireland and Europe, 
due to better di^nosis, prevention, and treatment. Instead, they had simply misattributed that trend 
around Dublin to effects of the coal-ban (Cox, 2012). This mistake was ultimately fixed in 2013, after the 
bans had already been extended, when the Health Effects Institute paid one of the original investigators to 
go back and consider control groups. Although methodologists and risk analysts had already noted years 
ago that the fact that both pollution levels and mortality rates have declined over time does not warrant an 
inference that reducing one reduces the other (Wittmaack 2007; Pelucchi et al., 2009; Cox, 2012), without 
access to the original data, they could not quickly and easily show that the original conclusions did not 
follow from the data. That had to wait until the original investigators were funded by HEI to try again 
more carefully. And by then, Irish public policy, based on a mistaken belief about the human health 
benefits to be expected fi’om extending the bans, had already been made (DECLG, 2012). 

We need not repeat such experiences here. We can choose to make the data available and to 
invite methodologists to take a look. Whether reducing current and recent past levels of air pollution 
should be expected to cause any reductions in mortality rates, and if so by how much, remains a great 
unanswered question - unanswered, that is, by sound science and statistical analysis of data. Today, 
answers are often simply assumed, without sound factual support, for purposes of regulatory benefits 
calculations (Cox, 2012, Chapter 7). It is possible and desirable to do much better. To do so requires 



42 


making original data open for others to analyze, and not to wait until policy has been made and changes 
enacted before allowing the public to find out whether better analyses would have led to different results. 
Thank you for your attention. 
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Chairman SCHWEIKERT. Thank you, Dr. Cox. 

To introduce our next witness I am going to turn to the Ranking 
Member Bonamici. 

Ms. Bonamici. Thank you, Mr. Chairman. 

I am pleased to introduce Dr. Ellen Silbergeld, a Professor of En- 
vironmental Health Science and Epidemiology at the Johns Hop- 
kins Bloomberg School of Public Health. Dr. Silbergeld holds a 
Ph.D. in environmental engineering, completed postdoctoral fellow- 
ship in environmental medicine and neurosciences, and has more 
than 40 years of scientific research experience in fields related to 
environmental health. She has been an appointed expert to the 
EPA, the Department of Energy, and many other federal agencies. 
She is a MacArthur Genius Fellow among her many honors. 

Thank you so much for being here to testify today. Dr. Silbergeld. 

Chairman ScHWEiKERT. Dr. Silbergeld, five minutes. 

TESTIMONY OF DR. ELLEN SILBERGELD, PROFESSOR, 
BLOOMBERG SCHOOL OF PUBLIC HEALTH, 

JOHNS HOPKINS UNIVERSITY 

Ms. Silbergeld. Thank you very much. I am appearing at your 
invitation to testify before the issues embodied in this bill and 
other issues that you have already alluded to, Mr. Chairman. And 
I have been a member, as indicated, of many expert panels in- 
volved in the evaluation of the scientific bases for regulation in the 
United States, the State of Maryland, and internationally. I also 
served as a member of the U.S. Delegation to the OECD during the 
development of the High Production Volume Chemicals Program 
which I would like to allude to. 

First, I want to join with you and others on this panel stating 
that the principles of openness and fairness are fundamental to 
science including toxicology, epidemiology, and basic research. And 
I agree with the statement of many at this hearing that there is 
an important need to reduce the secrecy that confounds public ac- 
cess to the basis for some EPA decisions specifically. However, with 
respect to my experience, the major driver of secrecy in EPA rule- 
making is the deference given to industry in terms of shielding its 
studies from public view, and thus I am puzzled as to the uneven 
nature of the debate on this topic and I hope that your Committee 
can see to that balance. 

The problem of nondisclosure by industry in fact was a key issue 
in developing the High Production Volume Chemicals Challenge 
Program by the OECD during the time that I was a member of the 
U.S. Delegation. And frankly, I have been very proud of the leader- 
ship role of American industry in the success of this program 
through which information held by industry was in fact made pub- 
licly available. And the current website of the American Chemistry 
Council makes clear that the industry shares justifiable pride in its 
disclosures and adherence to greater transparency data. 

We need more information, and specifically, we need more infor- 
mation disclosure by industry. Information withheld is not inform- 
ative. It — in fact, we can just look across the Potomac River to 
West Virginia and understand that if we had information, both the 
compulsion to produce it and to reveal it, how much better public 
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health authorities and civic authorities and the public itself could 
respond to that event. 

I would like to also draw upon my experience as an editor-in- 
chief of a major peer-reviewed journal and my experience over the 
past 18 years in terms of how science evaluates the quality of data 
that is published in the form of a scientific paper. The peer-review 
process requires the inclusion of scientific and technical informa- 
tion, including — as stated in your bill, sir — materials, data, and as- 
sociated protocols necessary to understand, assess, and extend con- 
clusions. The rest of the items in Section (2)(b)(3) of this bill, with 
respect, do not contribute to this goal in my opinion. 

We recognize that no study is perfect and frankly it is mostly 
protocol design and under-powering of studies rather than erro- 
neous statistical approaches that have resulted in withdrawal of 
many papers in my experience, and this is why in science we rely 
on replication as the means of validating the findings and conclu- 
sions of any particular study. But replication is not the same as 
data reanalysis. Replication involves the design and conduct of a 
wholly independent study often with different methods to test the 
reliability of the same hypothesis that was first studied. 

Let me also reflect on my experience with data analysis as part 
of the EPA’s process of reviewing science related to major regula- 
tion, as others have done on this panel. I was part of an expert 
panel advisory to the EPA under the Clean Air Act consideration 
of revising the National Ambient Air Quality Standards for lead. 
A reanalysis of the actual raw data was demanded by industry and 
it was accomplished in a nonadversarial way through third-party 
review undertaken by an acknowledged academic expert in bio- 
statistics not connected with government, industry, or the original 
investigators. 

In conclusion, I would like to restate my strong philosophical 
support for increasing the transparency of information associated 
with government regulation. I suggest that we already have the 
tools to accomplish this goal and through the implementation cer- 
tainly of the NIH covering data that is funded by that agency. I 
hope that your concerns can be reframed to apply to all sources of 
information in an effective and efficient manner because I know 
that some of my colleagues in industry have been vocal in calling 
for these steps. I call to them to tear down every wall — in the 
words of Ronald Reagan — that hides critically important informa- 
tion that is generated and held by industry. 

Thank you. 

[The prepared statement of Dr. Silbergeld follows:] 
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I am Ellen K Silbergeld, Professor of Environmental Health Sciences and Epidemiology at the 
Johns Hopkins Bloomberg School of Public Health. 1 am appearing at your invitation to testify 
on the issue of information disclosure and on the discussion draft entitled the “Secret Science 
Reform Act of 2014.” The views and opinions presented here are my own, and do not represent 
the views and opinions to the Johns Hopkins University. By way of background and experience, 
I have conducted research related to environmental health for over 40 years. I have also served 
on numerous expert panels, advisory boards, and as a consultant to the State of Maryland, 
National Research Council, EPA, DoE, CDC, FDA, NIH, NSF, WHO, ILO, UNIDO, FAO, and 
UNEP. Thus, I am familiar with the processes by which regulatory and scientific agencies 
identify and evaluate scientific information as part of the process of regulation. 1 was a member 
of several expert groups convened by EPA and the NRC considering the health impacts of lead 
in the environment, during which a re-evaluation of research data was undertaken. 1 have 
submitted my resume to the Committee in advance of this hearing. 

First, I want to state that the principles of openness and fairness are fundamental to science, 
including toxicology, epidemiology, and basic research. I have been a leader in the international 
movement towards adopting the principles of evidence based decision making in fields beyond 
clinical medicine and health care. I strongly support access to and sharing of scientific findings 
within the community of stakeholders in a manner consistent with principles of fairness and 
adherence to the goal of improving the process of decision making. These principles are 
described in my paper in ALTEX (attached). 

I agree with the statement of Chairman Smith that at present there is an important need to reduce 
the secrecy that confounds public access to the basis for some EPA decisions. In my experience, 
the major driver of secrecy in EPA rulemaking is the deference given to industry in terms of 
shielding its studies from public view. For this reason, I am puzzled as to the uneven nature of 
the debate on this topic, which we discussed in our commentary published in EHP (attached). 
The proposed bill would continue to immunize industry from disclosure while increasing the 
burden on EPA and, by pass through, on non-industry researchers. As noted in an earlier 
statement by Chairman Lamar Smith (November 2013), the interest of the public in the right to 
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see data is such high importance that the clouds of secrecy should be dispelled whatever the 
source. 

The problem of nondisclosure by industry was a key issue in the initiation of the High 
Production Volume Chemicals Challenge Program by the OECD (Organization for Economic 
Cooperation and Development). I was a member of the US delegation to the OECD 
Environment Program during the development of this voluntary process, which was initiated 
following a study by the Environmental Defense Fund (Toxic Ignorance 
http://www.edf.org/sites/default/files/243 toxicignorance O.pdf of which I was a coauthor). I 
was proud of the leadership role of American industry in the success of this program through 
participation in a tripartite partnership among government, industry and NGOs, to overcome the 
lack of basic toxicity data on most chemicals in commerce and consumer products. The HPV 
program has revealed that in many cases the critical data had already been generated but not 
released by industry. As stated by the American Chemistry Council on its website: 

Under the HPV Challenge Program, hundreds of chemical makers volunteered health 
and environmental information on 2,200 chemical products, representing approximately 
95 percent of the commercial market by volume in the United States, to help create 
a database that is available to the public. 

This voluntary initiative demonstrates that collaboration between public and private 
sectors can be an efficient method of developing safety information to help ensure the 
safety of the products of chemistry. 

With respect, this proposed legislation constitutes a retreat from this highly responsible and 
effective policy of information disclosure accepted and led by US industry. 

We need more information and more information disclosure by industry. Like trees falling 
unheard in the forest, information withheld is not informative. How much better would West 
Virginia have been able to respond last month if industry data were available and released on 4- 
methylcyclohexane methanol (MCHM) instead of the empty Material Safety Data Sheet: 
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Section III. 

Hazards Identification 

Acute Healtii Effects 

No specific infofnvjtiou av^aWs in our data base regarding the toxic effects of ihts materia for hunrans. HoAever, 
exposure to any cher«cai sNoutd be Kept to a nwimunt. Stan and eye contact n'>3y result in rrritalicxt. May be harmful if 
inhaled tw inge^ed. AIa' 3^ folto*' safe intfesIrM hy^ene practices and .year pre^r prcSective equipnient Atven handling 
this compound. 

Chrome Health Effects 

CARCINOGENIC EFFECTS : Not avaBa^. 

MUTAGENIC EFFECTS : Nest 

TERATOtSKtC EFFECTS : a-.u^aole. 

DENtLOPMENTAL TOXICITYMcl avaSabte 

Repeated or prolonged exposure to this is not kno ATS to aggravate exbtir^ ntedIcaJ conditions. 


As a scientist. I conclude that the broad sweep of stipulations in the draft bill is without a strong 
basis in terms of improving science or expanding the evidence base for decision making. I am 
also the editor in chief of a major peer reviewed Journal (Environmental Research) and in that 
role over the past 18 years I have considerable experience in and respect for the process of peer 
review as a method of quality assessment. The peer review process requires the inclusion of 
scientific and technical information including, as stated in the bill “materials, data, and 
associated protocols necessary to understand, assess, and extend conclusions.” The rest of the 
items in Section 2(b) (3) do not contribute to this goal, in my opinion. In science, we recognize 
that no study is perfect. That is why science has relied on replication as a means of validating 
the findings and conclusions of a particular study. “Replication” is not to be confused with data 
re-analysis; replication involves the design and conduct of a wholly independent study 
(sometimes with different methods) to test the same hypothesis. These are critical criteria for 
evidence in the standard methods of the Cochrane Collaboration. 

Let me also reflect on my experience of data re-analysis as part of the EPA’s process of 
reviewing the science related to associated lead as a risk for children’s neurobehavioral 
development relevant to the Clean Air Act. That re-analysis was demanded by industry and it 
was accomplished in a non-adversarial way through third party review undertaken by an 
acknowledged expert in biostatistics not connected with government, industry, or the original 
investigators. This review elicited some recommendations in terms of restating certain results 
but the main weight of the study was affirmed. And, of course, since that time, hundreds of 
independent studies have confirmed and extended the findings of that first publication. 

In conclusion, I restate my philosophical support for increasing the transparency of information 
associated with government regulation. I suggest that we already have the tools to accomplish 
this goal, in an even handed manner, through the methods of systematic review for evidence 
based decision making. I hope that your concerns can be reffamed to apply to all sources of 
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information in an effective and efficient manner. Given past history of contended regulations, as 
a scientist, an editor, and a citizen I am not convinced that the extraordinary and frankly arbitrary 
measures called for in this legislation will accomplish these goals. Because I know that some of 
my colleagues in industry have been vocal in calling for these steps, I would challenge them to 
tear down every wall, in the words of Ronald Reagan, that hides critically important information 
generated and held by industry. 

I am prepared to respond to your questions to the best of my knowledge. 
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Summary 

The concept of evidence-based toxicology (EBT) was proposed in 2006, but progress sinoe that time 
has been impeded by differing definitions and goals. This paper describes the parallels and discontinuities 
between the approach and methods of evidence-based medicine and health care and those proposed 
for toxicology. The critical element of an evidence-based approach for either discipline is the adoption 
of unbiased, transparent methodologies during the collection, appraisal, and pooling of evidence. This 
approach, implemented during the conduct of a systematic review, allows evaluation of the breadth 
and quality of available evidence. At present, systematic reviews are rarely done in toxicology by regulatory 
agencies, international organizations, or academic scientists. Adopting an EBT approach will necessitate 
significant changes in practice as well as attention to distinctive characteristics of toxicological studies, 
notably their emphasis on identifying harms and their reliance on experimental animal studies. An 
evidence-based approach does not obviate the role of judgment and values in decision making; its goal is to 
eitsure provision of all available information in a transparent and unbiased manner. 

Keywords: evidence-based toxicology, evidence-based medicine/health care, systematic reviews 


1 IntroducHon 

The concept of evidence-based toxicology (EBT) has been 
under discussion for several years (Hoffmann and Hanung, 
2006). EBT is about assembling the evidence related to hazards 
and risks of exposure, or to the evaluation of methodologies for 
assessing toxicology for the purpose of using this systemati- 
cally collected evidence during decision making. In this way it 
is simitar to Evidence-based Medicine and Health Care (EBM/ 
HC), which uses evidence derived from randomized control- 
led trials on which to base healthcare decisions. EBM/HC is 
defined as the application of systematically acquired evidence 
within the experience and expertise of the clinician, as well as 
patient values (Sackett et at., 1996). The essenfial premise is 
that decisions should be based on the evidence. It is important 
that the evidence be obtained in a transparent and systematic 
manner that is clearly described, enabling other investiga- 
tors to obtain the same evidence. Like EBM, the impetus for 
EBT clearly is related to the increasingly important role of the 
discipline of toxicology in decision making related to public 
health as well as clinical and prccHnical sciences. Progress in 


EBT has been impeded by differing ddinitions (Guzeiianetal., 
2005; Griesinger et al., 2009). both o( which advocate the use 
of methods developed for assessing and using evidence from 
randomized controlled trials for EB.M. an approach that is not 
feasible for the study of agents suspected of toxicity, as we will 
discuss below. Efforts also were impeded by a relatively lim- 
ited focus on the application of evidence-based approaches to 
the validation and acceptance of alicraauve methods in applied 
toxicology (Hariung. 2010). 

Evidencc-ba.sed decision making can be defined as the trans- 
lation of information into accepted practice using methods that 
reduce bias and increase confidence (Grimshaw et al., 2006). 
As in the law, evidence-based methods involve the evaluation 
of infonnation for its admission into consideration in decision 
making through the process of applying specified norms and 
methods. In order to avoid bias, these norms and methods must 
stand apart from the information under consideration, and their 
application must be undertaken with complete transparency. 

These characteristics differentiate evidence-based approach- 
es from current approaches used in the translation of toxicolog- 
ical studies into decision making by agencies concerned with 
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occupational and environmental health and consumer protec- 
tion, as we will demonstrate in this paper. In present practice, 
the identification of relevant primary studies and the norms by 
which these studies are evaluated in toxicology are largely im- 
plicit (the so-called Delphi method). As a result, the process 
clearly is not transparent and, because of this, it is difficult to 
avoid or reduce controversies over policy decisions incorpo- 
rating toxicology. A previous paper commented on the opaaly 
of the Delphi method often used in risk assessment (Silbergcld, 
2009), in terms similar to critiques of medical decision making 
using these methods (Rower et al., 2007). 

There is an understandable skepticism on the part of practi- 
tioners and experts in a field to the suggestion that the adoption 
of major changes in practice may be advantageous. This skepti- 
cism was expressed in the early days of EBM (Feinsiein, 1995; 
Williams and Gamer, 2002; Chalmers, 2005). We acknowledge 
and respect this natural skepticism in toxicology. This paper 
makes the case that adoption of evidence-based methods in toxi- 
cology may benefit from awareness of the history of evidence- 
based approaches in medicine and health care (EBM/HC). The 
goal of this paper is to introduce a consistent vocabulary for 
EBT and to examine the extent to which our experience in 
EBM /HC can inform the development of EBT. 

Al the outset, we recognize that it is reasonable to ask if 
adopting EBT will increase efficiency and quality of deci- 
sion making. The history of EBM/HC demonstrates that the 
evidence-based approach has accomplished these goals in 
medicine and many health care-related fields (Dickcrsin and 
Manheimcr, 1998). Moreover, this hisioiy shows that a com- 
mitment to an evidence-based approach in these fields has 
stimulated expansion and improvement in the field, specifi- 
cally through the development of systematic reviews as the in- 
strument for translating information into evidence. Systematic 
reviews often are considered the highest source of evidence in 
that primary studies are systematically identified and appraised 
and the totality of evidence is synthesized. This did not occur 
without considerable effort. When systematic reviews were 
initially conducted in medicine in the early ‘80s, many authors 
noted that methods associated with conducting systematic re- 
views were wanting in several areas, including repotting the 
primary studies, methods for identification and appraisal of 
the data, and methods for statistical pooling of the data (Mul- 
row, 1987; Oxman and Guyatt, 1988). The need to develop 
these approaches was not accepted readily by all practitioners 
(Chalmers, 2005). Nevertheless, over time, standards were de- 
veloped through consen.sus for reporting primary studies (c.g., 
the CONSORT statement and its extensions'), for reproduc- 
ibly searching for these studies (Dickcrsin ct al., 1994), and 
standardized methods to identify and account for biases in the 
primary studies (Moher et al., 1996). Also over time, further 
statistical methods and inferential models were put forward to 
synthesize similar research efforts. This focus on methods u.sed 
during the conduct of a systematic review process, in turn, has 


^ http://www.consort-statement.org/ 
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led both to greater transparency in reporting primary studies 
and to an increased focus on the quality of the studies compris- 
ing the evidence. 

Also of interest to the field of toxicology, the focus on study 
quality in t^M/HC, in turn, has influenced researchers in rel- 
evant fields to improve the quality of their research designs and 
the rigor of their statistical analyses in order to meet the cri- 
teria for inclusion in systematic reviews as well as to support 
evidence-based strategics. From the perspective of the devel- 
opment of toxicological sciences, this may be one of the most 
important benefits to consider in adopting EBT. 

There is concern that an evidence-based approach intro- 
duces rigidity into decision making (Gaichel and McGeary, 
2002) and through this may exclude valuable information 
through the use of scoring sy.stcms and meta-analysis. In an- 
.swer to these concerns, it should be noted that in EBM. HC the 
evidence provided by transparent systematic reviews prw ides 
only one stage of the evidence-based process of application 
of the evidence. This is not dissimilar to the role of toxicol- 
ogy in decision making as part of the overall process of risk 
management (NRC, 1994.) Any decisions made in EBMHC 
or toxicology must include consideration of other factors, such 
as cost, feasibility, and the bounds of accepted practice. Thus, 
in medicine, application of systematically acquired es idence is 
done taking into account the needs and values of the indiv idual 
seeking health care (Sackett et al., 1996). Moreos er. there is 
no requirement for evidence-based decision making to employ 
formal meta-analysis or to use forest plots to express iniegrai- 
ed findings.^ The use of systematic tools, when appn-^nate, 
is an important means of ensuring reproducibility of analssn, 
as well as the quality of the review, by ensuring comparaNlir) 
in design and conduct across the individual data sources, and. 
above all , enhanced transparency of conclusions reached in the 
systematic review. 

We argue that toxicologists should consider key lessons 
learned over the evolution of EBM/HC, First, such iransibons 
are best managed by the community of researchers aod pneti- 
tioners, rather than by imposition from outsiders (such as regu- 
lators and other consumers of toxicological evidence). Second, 
as demonstrated in current practice in EBM/HC, evidence- 
based methods do not reduce or replace the importance of ex- 
pert and experienced judgment. Rather, they simply provide the 
totality of evidence upon which to base those decisions. Third, 
the process in itself does not generate decisions. Simply put. 
an evidence-based approach assists the community by pro> id- 
ing systematically collected information using clearly described 
methods that reliably repre.sent the state of relevant knowledge. 
Thus, this approach assists decision makers in increasing the 
acceptability of their decisions by ensuring transparency dur- 
ing evidence collection. Fourth, a systematic and transparent 
approach to collecting and appraising the available evidence in 
EBM/HC has had a positive influence on researchers in terms of 
study design and data analysis. 
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2 Toxicology is not medicine or health care 

Despite the relevance of understanding the history and experi- 
ence in EBM/HC, there are characteristics of toxicology and 
its applications in public health that require more than simple 
adoption of EBM/HC methodologies. Some of these are related 
to differences in fundamental objectives. EBM/HC focuses pri- 
marily on developing evidence of the efficacy of therapy, to- 
gether with an emerging focus on the accuracy of diagnostic 
tests, as well as some focus on etiology, prognosis, and screen- 
ing. In contrast, the main focus of toxicology is on developing 
evidence for harms (hazard) and the magnitude or likelihood 
of hanns (risk). Although questions of harm have occasion- 
ally been the subject of EBM systematic reviews, as discussed 
below, many study designs utilized in generating evidence in 
EBM are not specifically intended to detect or characterize 
banns. Second, EBM/HC draws almost exclusively upon stud- 
ies conducted in humans and human populations; toxicology 
draws primarily upon studies conducted in nonhuman animals 
iu>d nonanimal models in order to achieve its societal goals of 
pro enting disease and disability. Thus it is important to recog- 
nize that adoption of evidence-based approaches for toxicology 
will require considerable work by the community, as discussed 
hdow. 


3 Assessing current proctice in toxicology 

To date, there have been relatively few explorations of the ap- 
piKanion of evidence-based practices to resolving issues of im- 
portance in toxicology. Toxicology has matured in the context 
of increased demands for its information through the growth of 
puMic concerns and regulation in environmental and occupa- 
tional health. The .structure of information needs for decision 
making in these domains of public health is relatively well de- 
fined lo include understanding the elements of relevant toxico- 
logical studies and major decision rules into which these 
ektnenis are to be incorporated. For the purposes of this pa- 
per. w e focus on those toxicological studies related to defining 
hazard and quantifying risk; exposure assessment, which is the 
other element of risk-based decision making, involves other dis- 
ciplines and methodologies. Hazard and risk are common to the 
practice of risk assessment and to application of the precaution- 
aiy principle, which has been advanced as a partial alternative 
to risk assessment based methods related primarily to reducing 
the burden of infonnalion required for undertaking assessments 
(Silbergeldetal.,2004). 

Current evaluations of toxicologicd information (from hu- 
man and nonhuman subjects), almost without exception, have 
failed to utilize systematic or transparent methods. These limi- 
tations are exemplified by a review on lead and cancer by one 
author of this paper (Siibergeld, 2003) and a review of the car- 
cinogenicity of lead compounds by the International Agency 
for Research on Cancer (lARC, 2006). Both of these examples 
are distinguished by lack of transparency such that it is not 
possible to determine or to replicate the process of identify- 


ing studies or their selection for review. No information w'as 
provided on the search strategy or on screening criteria in terms 
of study quality. Without this information, it is not possible to 
ascertain the completeness of the review. There is no disclosure 
of which studies were discarded or why they were discarded. 
Further, there is no information on why certain studies were 
emphasized in the discussion. In the case of experimental stud- 
ies, a similar lack of transparency informed the identification 
and selection of studies, A recent comment on the failure of 
lARC monographs to utilize systematic approaches or to cite 
systematic reviews echoed these same concerns with additional 
examples (Straif et al., 20 1 2). 

In these two c.xamples, the review of epidemiological stud- 
ies combined cross-sectional, longitudinal, cohort, or second- 
ary analyses without acknowledgement or discussion of het- 
erogeneity, even though it was unlikely that their results could 
be combined in any meaningful manner. Similarly, the in vitro 
studies were discussed without consideration of study design, 
dose or in vitro concentration, animal strain or cell line. Other 
sources of heterogeneity were obvious as well. Sometimes stud- 
ies actually reported on different endpoints. These problems are 
increased when multiple experimental tests are used to define 
an endpoint, such as multiple in vitro systems and different ani- 
mal strains (for example, in current US EPA guidelines for de- 
velopmental neurotoxicity (Croflon ei a!., 2004) and endocrine 
disruption (Daston et a!., 2003)). When the methods of such 
studies are so diverse, it may not be appropriate to combine 
results except in the most general way. Similarly, in EBM/HC 
studies arc not combined if they show either clinical or statisti- 
cal heterogeneity. 

In place of a formal integration of results using clearly de- 
scribed methods (c.g., formal meta-analyses or focused narra- 
tive syntheses of the data), these reviews included only tables 
that summarize selected findings. The only qualified Judgments 
relate to carcinogenicity using EPA or lARC criteria. Even 
more disturbing than these examples is the practice in some 
health assessments to base conclusions on only a few or even 
one study, judged to be the most appropriate or reliable (on 
nontransparent criteria). Facing two alternative conclusions, 
one must “choose” which one, if either, to believe. In contrast, 
a systematic approach uses all the accepted evidence on which 
to provide a basis for decision making. The concept of a “key 
study” is contrary to the notion of a systematic review because 
of its deliberate exclusion of the body of relevant information. 
This selective practice was followed in a recent NRC review 
of mercury, in which a nontransparent decision was made to 
reject one of two large prospective epidemiological studies on 
early exposure.s to methyl mercury and neurodevelopmental 
outcomes (NRC, 2001). Another approach on this same topic 
utilized a self-described Bayesian “integrative” approach to ex- 
amine several studies, but no reason was provided for why only 
some pertinent studies w’ere included (Axelrad et al., 2007). 
The recent NTP review of lead (2011) moves closer to the prac- 
tice of systematic reviews as practiced using an evidence-based 
approach, but it is .still a mixture of transparent and nontrans- 
parent methods. There are clear statements related to framing 
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specific questions and to some extent explicating the initial cri- 
teria for searching the literature for relevant primary studies, 
but it fails to present an explicit means by which these studies 
were identified or evaluated. In addition, as stated in the report, 
NTP explicitly relied upon other “authoritative sources” (from 
US government agencies) to identify citations for review, sup- 
plemented by some searches of the literature and consultation 
of experts rather than systematically reviewing all relevant cita- 
tions. Thus, it is difficult overall to define the methods by which 
the primary studies were identified or selected, and it is likely to 
be difficult to replicate the process in an independent exercise. 
Most importaniiy. the document does not describe how these 
study results were integrated to support qualitative judgments 
based on lARC criteria. Tables in the document are rated as 
either '’supponing” or “not supporting” these qualitative Judg- 
ments without defining or describing the criteria used to classi- 
fy a study as supporting or not. Furthermore, the authors appear 
to have selected which studies are cited in these tables rather 
than showing all data. Evaluation also involved nontransparent 
processes such as expert consultation and review by a selected 
panel. The conclusions were further influenced by the commit- 
tee review, as welt as by the conclusions of the “authoritative 
sources.” which, as noted above, did not adopt or implement 
transparency. 


4 Why BT end vtdiy now 

The need for EKT is arguably driven by several forces: the in- 
creased demand for transparency and a stronger scientific basis 
for decision making in both public and private sectors, as well 
as longstanding dissatisfaction with the pace and contentious 
nature of current inodes of decision making in public health 
(EEA, 2001). Examples such as the divergent risk assessments 
for methyl menniry and bisphenol A in public health policy in 
the US and the El- 1 Beronius ei al .,2010) do not encourage con- 
fidence. Stakebc4dcrs with an interest in efficient government 
and public health should be greatly concerned by the fact that 
F-PA'.s evaluaiiofl of the human health effects of dioxins took 18 
years. How the data used to make these decisions vvas obtained 
is neither clear nor replicable. EBT mandates the provision of 
methods used to develop a set of primary studies which are then 
used as the evidence for decision making. Clearly, the use of 
EBT can promote reduction of controversies, as all can obtain 
exactly the same data on which to base decisions; the meth- 
ods used to obtain, assess, and integrate the data are described 
clearly enough to allow replication. In addition, through in- 
creasing the efficiency of decision making, EBT can respond 
to societal pressure to decrease the resources of time, money, 
and vertebrate animals utilized in reaching decisions related to 
hazard and risk (Rovida and Hartung, 2009). These pressures 
have increased interest in developing alternative methods that 
reduce the time required to obtain relevant information (NRC, 
2007). For this reason, the need to validate these alternative 
methods adds further impetus to EBT. 


5 Initial steps towards systematic reviews 
in toxicology 

We have carried out some of the more detailed studies using 
principles of EBM/HC to evaluate the evidence for associations 
between environmental toxicants and human health risks, and 
this experience provides some perspective on the challenges in 
adopting and adapting these methods to EBT (Navas-Acien el 
al.,2005, 2006, 2007; Maul! et al., 2012). These reviews follow 
the norms of transparency and methods that have been devel- 
oped for systematic reviews of diagnostics and interventions in 
medicine and health care. They incorporate the following steps; 
development and explicit framing of research questions that 
can be answered by a systematic review plus explicit statement 
of a publically available protocol for conducting the systematic 
review. This protocol includes a defined and annotated strat- 
egy for locating sources of evidence; a priori conditions for 
exclusion and inclusion; defined analytic procedures to evalu- 
ate study designs and statistical methods; criteria for evaluating 
selected studies; methods for integrating study results. These 
rules are based on the assumption that all studies are well in- 
tentioned but no study is perfect. The goal is to identify ail rel- 
evant sources of information in an unbiased manner and then to 
screen this body of information by identifying aspects of each 
study that can increase bias or uncertainty and to consider the 
impact of these aspects on analytic confidence. 

Our attempts to integrate toxicological studies into our re- 
views were limited in tenns of availability of studies, due in 
most cases to the variability in study design or in the endpoints 
selected, as well as to differences or lack of precise informa- 
tion on dosing and dose duration, and uncertainty as to the rel- 
evance of measured outcomes to the inference of human health 
risk. Some of these issues relate to toxicology, in which a range 
of endpoints often are utilized as relevant indicators of human 
disease risk; this is related to the lack of accepted phenotypic 
animal (or in vitro) models for many human health endpoints 
and uncertainty as to mechanisms involved in human disease. 
Lacking a coherent nosology, toxicological studies are likely to 
be more varied in design and endpoint than epidemiological or 
clinical studies. Integration of different endpoints may he pos- 
sible using a systems biology approach to group endpoints in 
terms of common pathways, but this has not been tested in prac- 
tice. These concerns also were cited by Maull et al. (2012). 

Asimilarexperienceis presented in an excellent recent system- 
atic review of formaldehyde and reproductive and development 
endpoints (Duong et al., 2011). The review of epidemiological 
studies is a model in transparency and rigor. In contrast (and 
similar to our reviews on lead and arsenic mentioned above), 
the review of experimental animal studies vvas less transparent. 
No clear information is presented on search terms and criteria 
for inclusion or exclusion of studies. Large differences were 
noted among studies in terms of .species, routes of exposure and 
dose, as well as endpoints, which probably impeded any attempt 
at integration such that only a summary of “key findings” was 
presented. A thorough narrative discussion of mechanisms and 
modes of action also was included. 
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6 Challenges for EBT 

The results of our analyses, along with more recent experience 
from an expert working group convened by the Naticmal Insti- 
tute of Environmental Health Sciences (NIEHS) to evaluate 
associations between environmental chemicals and dlab^es, 
indicate that toxicologists have considerable work to do to im- 
plement an evidence-based approach (Silbergcld, 2009). Inno- 
vations and modifications are especially needed to develop ev- 
idence-based methods tailored for toxicology and experimental 
nonhuman studies. Some of the major limitations noted in our 
reviews are discussed here for human studies and experimental 
studies. First, the amount of primary' information available from 
independently conducted epidemiological studies in the pub- 
lished literature is relatively sparse for many exposures of inter- 
est. Second, many of the available epidemiological studies have 
significant problems in terms of study design or data reporting 
.such that it is difficult to identify biases in them. For e.xample, 
in many studies of arsenic, there are limited or r>o data on in- 
dividual exposures and many studies failed to collecl or report 
infonnation on important covariales and currfounders or infor- 
mation sufficient to determine helerogcncity. Many studies arc 
relatively small and likely underpowered; many of the studies of 
larger cohorts (such as NHANES) are not aoually independent 
of each other, and none are longitudinal, and so caisality cannot 
be inferred in terms of exposure preceding outcome. In addi- 
tion, there are broad differences in definition and measurement 
of outcomes of interest. This is understandable for toxicologi- 
cal studies, but is also characterisik of marry eptdemtoiogical 
studies on, for example, lead and arsemc. For the toxicological 
studies, there is enonnous heterogeneity in all aspects of study 
design and interpretation, as discussed above arkd in Duong et 
al. (2011). These criticisms were similar to the evaluations of 
the medical literature in the early "SOs when systematic reviews 
in EBM/HC were first widely applied and just beginning to be 
appreciated (Dickersin and Manheimcr. 

Nevertheless, our reviews demonstmed that important el- 
ements of the methodology of systemauc rcMcws can be 
adopted by EBT with little change, notably an allegiance to 
transparency in methods for searching the available literature 
for potential evidence, in selecting studies for rev iew, and ap- 
plication of a priori criteria for assessing each selected study. 
Toxicologists can examine existing criteria for systematic re- 
views of observational epidemiology (Biair eial.. 1995; AMS, 
2007; Longnecker et ai., 1988). When appropriate, some of 
the methods for integrating results across studies also may be 
adopted. From our analyses, we also observ'ed that the greatest 
challenges for developing EBT are related to handling infor- 
mation from experimental nonhuman studies, where there is 
no consensus on analytic procedures and where even the con- 
struction of research questions may be more complex owing 
to the many test systems and endpoints used in studies on the 
same topic. In addition, there is no consensus on methods for 
screening primary studies, for evaluating the selected studies. 


3 http://www.cochfane.org/resources/handbook/ind 0 x.htm 


and on appropriate statistical methods to integrate study results 
from the range of experimental designs. This challenge will not 
be met by selecting information only from standard toxicol- 
ogy test guidelines or Good Laboratory Practice requirements 
as the definition of acceptability for evidence-based decisions. 
Many of these designs are extremely limited and, while they 
may produce data of use in standard risk assessment methods, 
they are underpowered and not robust (Reuter et al., 2003). As 
has been noted in endocrine disrupter research, these types of 
studies may be less informative than research studies that are 
more specifically designed to investigate defined hypotheses 
rather than to generate minimal information on hazard (Myers 
el al., 2009). Rather, all relevant studies should be sought and 
then evaluated using methods for appraising sources of biases 
identified through a consensus process in order to determine the 
strength of the evidence provided by each. Achieving this goal 
will foster a closer relationship between environmental epide- 
miology and experimental research, going beyond the invoca- 
tion of experimental research merely to satisfy one of Bradford 
Hill’s recommendations. 

Achieving the goals of evidence-based and systematic analy- 
sis, as argued by practitioners in EBM/HC, has involved two 
strategies implemented at the beginning: involvement of a 
broadly based community for achieving consensus in methods 
and evaluations and a commitment to complete transparency. 
These commitments are e.xempltfied within the Cochraue Col- 
laboration. At its inception, the Collaboration included only a 
few dedicated investigators with a shared vision to help people 
make good health care decisions. This goal drove the devel- 
opment of systematic reviews and the dissemination of these 
reviews, which now cover a broad range of topics related to 
health care inien’enlions. Key principles of transparency and 
continuou.s improvement in methods based on empirical evi- 
dence underlie the growth of the Cochrane Collaboration and its 
influence in the field of EBM/HC.^ This paper argues that these 
slrate^es, as well as a commitment to continuous growth and 
improvement in methods, are equally critical for the successful 
development and adoption of EBT. 

The decision for EBT involves a commitment by the field 
of toxicology, not only to science but to community. As not- 
ed above, practitioners in EBM/HC stre.ss that iLs .success Itas 
invdved the engagement of a broadly based community for 
consensus evaluations and a commitment to complete transpar- 
ency. These steps cannot be rushed by establishing structural 
frameworks and empty institutions but must be grown from an 
organic discussion among the community of stakeholders, in- 
cluding scientists, technicians, governments, private sector, and 
the public (Chalmers, 2005). 

Our success may transfonn the field of toxicology, as well 
as (he practice of decision making in regulation. EBT can con- 
tribute to the efficient adoption of aiiernalive methods through 
consensus agreement on identifying the evidence and on criteria 
for evaluation, drawing on experience from diagnostic evalu- 
ations in EBM/HC. However, there must be a commitment to 
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empirically testing the methodology for systematic reviews of 
toxicological data; without such methodological studies, the 
field cannot move forward. This will not be a simple task. Since 
toxicology is fundamentally a science of prevention (Siibergdd 
et al., 2004), its aim is to detect likely harms prior to human 
exposure. For this purpose, experimental studies are the only 
source of truly preventive information, and thus the focus of 
EBT should be on experimental toxicology and test methods in 
the broadest sense. 

Adoption of an evidence-based approach does not mean the 
adoption of the clinical trial design as the “highest” or only 
form of reliable infoimauon (Silbergeld, 2009). Evidence may- 
come from any type of study, and although many reviews focus 
on randomized clinical trials, the type of evidence (i.e., study 
design) required depends on the type of research question (e.g., 
the use of randomized controlled clinical trials to answer ques- 
tions of efficacy and cohort or case-control studies to answer 
questions related to euology). This has facilitated the develop- 
ment of both “rules of practice” and the post hoc evaluation 
of research results (Dickersin and Manhcimer, 1998). EBM/ 
HC also provides a rich source of valuable guidance to EBT in 
its methods for evaluating obseix^ational epidemiology fBlair 
et al., 1995; Longnecker et al., 1988; AMS, 2007). While we 
can learn from EBM/HC, as noted at the outset of this paper, 
the issues of concern to toxicology, for the most part, are noi 
the same as those in medicine and health care. In EBM.HC. 
the evidence-based approach has been developed mo« fully 
for answering questions related to therapy and diagnosis. The 
evaluation of novel test methods {such as aliemauve syuemsi 
may draw usefully upon methods used iu evaluating diagm^ 
tics. Systematic reviews using only evidence from randofm/ed 
controlled trials (RCTs) are not well suited to identifying 
harms, primarily due to study designs focused on identifying 
benefit, often with insufficient power to delect adverse effects 
because of the relatively low number of individuals exposed 
and the short time frame of many RCTs (Chou and Hdfand. 
2005). 

The investment of our community in developing EBT will 
be worthwhile. In the absence of an evidence-based process, 
decision making is dependent upon a pseudo-Delphi process, 
in which experts are convened to undertake a qualitative proc- 
ess of integrating and weighing information (e.g.. the NTPand 
IARC).This is less and less satisfactory to the public and other 
stakeholders; it is also highly resource-intensive in terms of 
repetitious studies atid expert consultation (Rovida and Har- 
tung, 2009). EBT will lead us into new domains of science and 
assessment, but we should remember tliat.in identifying harms 
and assessing risks, as in the law. an evidence-based approach 
does not remove the need for the application of judgment 
(Sackeit etal., 1996; NRC, 1994). The premise and promise of 
EBT is the reduction of uncertainties by assuring a consistent 
body of information and enhancing confidence in the selection 
and evaluation of this information through a fully transparent 
process dedicated to continuous improvement through experi- 
ence. These were the goals that inspired Archie Cochrane and 
the early community of analysis; by adopting them, trie com- 


munity of toxicologists can enhance the developmetit of sci- 
ence and better serve the social goals of health protection and 

safety assurance. 
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The U.S. Environmenral Protection Agency 
(EPA) is one among many agencies covered 
by the Information Qualify Act (IQA 2001), 
an amendment to the Treasury and General 
Government Appropriations Act for fiscal year 
2001 that has Ixen viewed as a mechanism to 
increase access to such information and to seek 
corrections If parties think that government 
agencies have used faulty information and 
analyses. Tlve Office of Management and 
Budget (0MB) issued IQA guidelines that 
apply to all agencies in the Executive Branch: 
When these agencies provide “influential scien- 
tific, financial, or statistical information," they 
also “shall include a high degree of transparency 
about data and methods to facilitate the repto- 
ducibiliry of such information by qualified 
third parties” (OMB 2002). The law was 
enacted without debate or hearing. In the 
absence of an extensive legislative history and 
because both the IQA and OMB guidelines 
were silent about whether agency responses 
were judicially reviewable, some bad viewed 
the act as providing a new avenue for legal 
challenges of agency decisions across the U.S. 
government. For example, in 2006 the U.S. 
Fourth Circuit Court of Appeals ruled that 
plaintiffs did not have standing to sue the 
Department of Health and Human Services 
under Title III of the IQA to compel access 
to a study conducted by the National Heart, 
Lung, and Blood Insriturc (NHLBI) that 


was used to support action by the Food and 
Drug Administration (FDA) on dietary salt 
(Salt Institute v. Leavitt 2006). A number of 
industry groups had petitioned the NHLBI to 
make the raw dau from the study available so 
that they could do subgroup reanalyses. The 
cuun found chat the plainttifs had received no 
injury from being denied access to the NHLBI 
data and thus did not have standing. However, 
the court also noted that the petitioners had a 
longstanding right to request the raw data Ifom 
the study using the Freedom of Information 
Act (FOIA 1966). In its response, the NHLBI 
noted that it was preparing a public access data 
set for release, which it later made available 
(NHLBI 2005). Although this case was 
resolved under existing FOIA mechanisms, 
in the wake of this litigation there has been 
concern rhat the IQA docs not provide outside 
panics suffidenr access to die data for studies 
that underlie regulatory decisions made by 
U.S. govemment agendes. There is increasing 
interest in improving the methods by which 
chemical and pesticide hazards and risks are 
evaluated not only by government but also 
by independent scientists (Bucher et al 2011; 
Woodruff et al. 2011). This interest has 
spurred increased demand for transparency and 
disdosure of the data used by the U.S. EPA 
to make evaluations that support r^ulatory 
decisions for chemicals and pestiddes. In this 
context, we examine the tole of the IQA in 


making such data more accessible and surest 
alternative approaches. 

Review of Requests for Data 

To find out how responsive the U.S. EPA 
has been to requests for raw data under the 
IQA, wc reviewed 79 requests filed with the 
U.S. EPA between 2002 and 2012 either 
to correct or to reconsider the data that the 
U.S. EPA used in evaluations supporting its 
regulatory decisions during that period. Under 
OMB guidance for the IQA (OMB 2002), 
parties cm request that agendes reconsider 
or correct any information used to support 
regulatory decisions; usually these requests 
are made in the ft>nn of leners. The U.S. EPA 
posted these 79 requests on its web site, 
according to OMB guidelines (U.S. EPA 
2012a). Interestingly, only two of these 
requested raw data. 

The Tint request for raw data was filed 
in December 2003 by the Perchlorate Study 
Group, an indusrey consortium of manu- 
facturers and usen of perchlorate (Aerojet, 
American Pacific Corporation, Kerr-McGee 
Chemical, and Lockheed Martin). They 
requested riui the U.S. EPA provide raw data 
from expqitixmal snidies (Girard 2003). Hie 
U.S. EPA granted this request in September 
2004 and provided aces to brain images and 
contranor's reports (Gilman 2003). 

The second case was filed by the Association 
of Battery Recyden (ABR) in October 2008 
(Steinwumtl 2008). Now called America’s 
Battery Recyclers, and formerly called the 
Secondary Load Smekers Association, the 
ABR is a group of auto and industrial battery 
recyclers, primary lead producers, and users 
of recycled lead {America's Battery Recyclers 
2012). The ABR requested raw data from a 
study of lead toxidty (Lanphear et al. 2005) 
that was among several published studies relied 
upon by the U.S. EPA in its development of 
the National Ambient Air Quality Standard 
(NAAQS) for lead under the Clean Air Act 
Amendments (1990). Because the ABR and 
others had taken the U.S. EPA to court to 
overturn the lead NAAQ rule at the same 
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time, the U.S. EPA opted to postpone consid- 
eration of the request under the IQA pending 
the decision of the court. In its response to 
the request, the U.S. EPA noted that con- 
cerns about the data analysis had been noted 
in comments during the rule-making process 
and that the U.S. EPA had commissioned new 
external peer reviews of the study (U.S. EPA 
2012a) in addition to a reanalysis of the data of 
Lanphear et al. (Rothenberg and Rochenberg 

2005) . After the lead NAAQS was upheld 
in July 2010, the ABR again requested that 
the U.S. EPA provide access to the Lanphear 
data (Steinwurtze! 2010). Meanwhile, litiga- 
tion was filed over the delay in providing the 
data. This litigation was dropped when the 

U. S. EPA POIA office worked our an agree- 
ment with the Cincinnati Children’s Medical 
Center ro obtain the Lanphear srudy data 
(Lanphear DP, personal communication; Poh! 

V. U.S. EPA et al. 2012). U.S. EPA attor- 
neys determined that access to the data was 
required under rhe 1998 Shelby Amendmcnr, 
which makes federally funded research data 
accessible to the public under FOIA (Treasury 
and General Government Appropriations 
Appropriations Act 1998). Thus, as for the 
request to the NHLBI to provide data con- 
cerning the salt srudy (Salt Institute v. Leavitt 

2006) , the resolution of the request was man- 
aged under FOIA. 

Because requests for raw data are few 
and fer between, it has not been onerous for 
the U.S. EPA to provide such data. Existing 
mechanisms have provided the ability to reana- 
lyze data by a) development and av^abilicy of 
a public-access database (with suitable protec- 
tions for the human subjects involved in such 
studies); b) provision of raw data via FOIA, for 
cases in which data are In possession of or cm 
be obtained by the agency (e.g,, the perchlorate 
case cited above); and c) rcanalysis of data by a 
third party. As an exampie of the rhitd mecha- 
nism, the widely publicized results from the 
Harvard Six Cities Study (Dockety cc al. 1 993) 
were used by the U.S. EPA in 1997 as a basis 
for developing new standards for fine particu- 
late matter (S 2.5 pm in aerodynamic diam- 
eter) ait pollution (U.S. EPA 1997). Interested 
panics, mostly from industry, raised questions 
about study analysis and interpretation. The 
raw data were nor in the possession of the 
U.S. EPA, and tbe U.S. EPA could nor compel 
rhe submission of these data from Harvard 
University or the funding source, the American 
Cancer Society. Under pressure from govern- 
ment agencies and industry. Harvard and the 
American Cancer Society voluntarily requested 
that the Health Effcas Institute (HEI) step in 
as a third party to supervise a rcanalysis of their 
data. "Ihe HEI [a consortium of industry, aca- 
demic, and government scientists established 
by the Clean Air Act Amendments (1990)} 
provided the data for reanalysis by a third party 


selected by a sdence advisoiy committee that 
included representation from Interested parries 
who had argued for an independent reanalysis, 
thus providing a process to address the uncer- 
tainties about the analysis and interpretation 
(HEI 2000). 

Discussion 

Over lime, the U.S. EPA has come to rely 
incteasingly on a large number of scien- 
tific studies to compittc reviews for a single 
chemical. This is illustrated by the case of 
2,3,7,8-TCDD (2,3,7,8-retrachlorodibenzo- 
^dioxin). In its recent assessment of TCDD, 
the U.S. EPA identified some 2,000 studies 
directly relevant to its review of dioxin Doxicity. 
From these, the U.S. EPA selected 10 “key” 
epidemiologic studies and 74 “key” experi- 
mental animal studies. Even for this smaller 
subset of “key” studies, the raw data for each 
human study and animal experiment are sub- 
stantial, ^d most of the data on TCDD were 
nor in rhe possession of the U.S. EPA (2012b). 

There are several mechanisms through 
which the U.S. EPA might obrain these data. 
The U.S. EPA could requite that investiga- 
tors submit ffieir raw data ro rhe agen<y upon 
complerion of rheir research as a condition 
of U.S. EPA funding, but this would not 
completely solve the problem. Most research 
evaluated by the U.S. EPA for regulatory deci- 
sion making is not funded by the U.S. EPA. 
In these cases, the U.S. EPA would have ro 
undertake an extensive collection of raw data 
from study investigators, which would be 
costly ro the U.S. EPA and burdensome to 
the research community. Nor insignificandy, 
this would create major delays in rule mak- 
ing. In terms of resource allocation, it is rea- 
sonable to ask how much of rhe U.S. EPA’s 
budget could be allocated ro accomplish this, 
and where this would rank relative to other 
priorities, such as increasing the numbers of 
priority assessments to mcer the U.S. EPA’s 
statutory goals. 

In addition to the burden on the U.S. EPA, 
there wuuld be a significant burden on the 
scientific community that produces most of 
the relevanr research, and it is very likely that 
there would be significant pushback from the 
academic community under rhe Paperwork 
Keduaion Act (1995). In feet, the Paperwork 
Reduction Act, which was enaaed to reduce 
the total amount of paperwork handled by 
the U.S. government, would not allow the 
U.S. EPA to undertake such a massive data 
colleaion without establishing that the bur- 
den imposed upon the research community 
would be justified by the benefits of providing 
the data. 

At the least, scientists would need funding 
to respond to requests that are generated as 
a consequence of the use of their studies by 
the U.S. EPA rather chan any action taken 


by the investigators themselves. Burdened by 
other responsibilities and unable to fond such 
activities from grants provided by sources 
other than the U.S. EPA, scientists are not 
likely to voluntarily provide rhe U.S. EPA 
with raw data from studies conducted months 
to decades in the past simply because the 
U.S. EPA has decided to include those studies 
in their latest assessment. 

Moreover, the U.S. EPA would not have 
clear legal authority to compel rhe submis- 
sion of data from industry, federally funded 
studies conducted prior to the 1998 Shelby 
Amendment, studies fonded by other federal 
agencies, or studies that are not fonded by 
the U.S. government, including studies from 
non-U.S. investigators. We therefore con- 
clude that a regulatory approach, in which 
the U.S. EPA compels the submission of raw 
data for all studies reviewed for rule making 
on pesticides and chemicals, would not be 
tenable. It could in fact have a chilling effect 
on the engagement of the global scientific 
community in research relevant to the protec- 
tion of human health and the environment. 
Certainly, this is not in the best interests of 
sciencc'ba.scd policy. 

In addition, there are other fcasibiliry 
issues. In the case of older studies, raw data 
may not cxfrr or may be difficult to aeccu 
because of storage on outdared media sujffi as 
tapes. For epidemiologic .studies, considen- 
rion would need to be given ro nhkal issuo 
governing studies of human subjects. These 
include protection of coiifidenrialiiv and pn- 
vacy, and prevention of abuse of the dao. for 
example, by marketing companies who may 
wish to identify patients with particular medi- 
cal conditions. Ginical-crials invesiigaion have 
been working for years to develop wa>-s to 
disclose data from human studies, including 
mechanisms for placing data behind a bar- 
tier to universal access, so that it is accessible 
tinly to those who meet conditions of use. In 
the case of clinical trials, there arc studies in 
which removal of all identifying data negates 
its scientific value; therefore access to the data 
would need to be limited to protect privacy 
(Hfynaszkiewicz et al. 2010). Vi^th adequate 
resources and planning, these obstacles could 
be anticipated and/or overcome. 

In rhe case of research data concerning 
chemicals and pesticides, the U.S. EPA also 
is constrained by legal constructs that have 
defined regulatory resting of pesticides as 
“confidential business information” (CBI) 
and that require the U.S. EPA ro redact cer- 
tain data and obtain affirmations from recipi- 
ents that they will not give the remaining 
data to multinational companies that might 
seek to register rhe pesticide to market k in 
other countries (U.S. EPA Office of Pesticide 
Programs 2010). The U.S. EPA could improve 
the web access ro summaries and analyses of 


ISO 


vauMi 1211 NUMBER 2 i February 2013 • Environmental Health Perep^ives 



60 


Assuring access to data for chemical evaluations 



these data, which are publically available but 
often difficult to find in web searches. This 
would not be the same as providing access 
to raw data. We therefore suggest that, in 
the short run, industry should work with the 
U.S. EPA to identify approaches to provide 
more robust data sets for studies that they 
submit to the U.S. EPA. lire U.S. EPA also 
could invite companies to voluntarily waive 
CBI claims on re.sts of pesticides and chemi- 
cals. In the long run, we think that Congress 
should amend the Toxic Substances Control 
Act (1976) and the Federal Insecticide, 
Fungicide, and Rodenticide Act (1972) as 
amended by the Food Quality Protection Act 
(1996) to ease CBI protections from pesticide 
and chemical test data. 

In an ideal world we would always favor 
more disclosure over less, but it b not clear 
how this should be done, or who should pay 
for it. The HEI, which has an independent 
governing board and is supported by a con- 
sonium of funders including the U.S. EPA 
and the automobile and petrochemical indus- 
tries, may be a useful precedent. The HEI 
requires that data from all HEl-frmdcd studies 
be made as'ailabte as expeditiously as possible, 
[lakinjJ into consideration the legitimate intei- 
Icctiui iatemn of the investigator to have the 
iif<p>ftun«y to hcncdc from his or her Intciicaual 
enjc(*on and to publish snbscquent analyses from 
the ebu wi iawioding additional analyses frinded 
brHEI;.iHF.I.10ia) 

The HEI anempts to balance the interests 
of imTstipion with those of interested par- 
ties in cases of ‘studies of particularly high 
regulatory importance being used to inform 
decisions over a shotr time frame,” and 
encour^es ks principal investigatot.s to share 
the daa ouarpr in situations where “providing 
the data wxKiid place an undue burden on the 
investigator' (HEI 2010). For example, in 
«-hcn there have been so many requests 
that it was difficult for the investigators to 
continue their racarch, the HEI has assisted 
investigators with data sharing. In addition, 
the HEI requires that data requesters pro- 
vide “reasonable reimbursement for both the 
direct costs of providing the data, and for the 
time of the investigator and/or HEI staff to 
gather, transmit, and explicate the data” (HEI 
2010). HEI also “will consider requests from 
the investigator for a reasonable budget of 
data archiving funds, to be provided as pan 
of the project budget” (HEI 2010). From 
this precedent, it seems that proponents of 
increased access to raw data need to consider 
not only frnandal and time burdens on inves- 
tigators, but also a way to reasonably balance 
the need for data access with the ability of 
investigators to realize the fruits of their own 
intellectual endeavors. 

Another useful precedent that could serve 
as a model for data sharing is the National 


Institutes of Health (NIH) clinical trials 
database (CUnicaiTriak.gov; NIH 2012). It 
does not contain “raw dara” but rather con- 
tains detailed and useful information about 
dinical-tiial stutfy dedgns and stamdes chat not 
only convey results in a standardized fashion 
but also identify imporont quality par^eters 
(e.g., drop-oui rates). Required by law (Section 
113 of the Food and Drug Administration 
Modernization Act 1997), the clinical trials 
database was developed by the NIH with input 
from the FDA and the National Library of 
Medicine (NLM). Currently, many medi- 
cal journals require chat triak he rcgisceretl in 
CUnicarrriak.gov prior to their publication; 
as of 3 December 2012, 136,605 studies in 
182 countries were rostered. Althou^ many 
rescarchm are now arc calling for access to raw 
data for all clinical triak (Gorzsche 2011), the 
ClinicaiTriak.gov database has gready increased 
access to information about drug efUcaqr triak 
and drug safety, and the development of sudi a 
database for studies of chcmicak ^d pesticides 
would he a major step toward increasing the 
transparency of the U.S. EPA’s evaluations and 
making data more accessible to third panics. 

Conclusions 

At present, there does not seem to be a large 
demand for raw data related to U.S. EPA 
decision making; however, this may change 
as format evidentiary reviews of environ- 
mental health re.search become increasingly 
common (Maull ct al. 2012). Compared with 
clinical triak, the acquisition of raw data for 
chemicals and pesticides would be much more 
complex, in parr because it would require a 
framework chat can accommodate data from 
numerous types of studies: observational and 
experimental, animal, human, in vitro, and 
hi^i throughput screening studies. 

For human epidemiologic studies, clear 
and complete documentation would need to 
be provided for interpretation of the variables 
collected in such studies. This is no simple task 
given, for example, a) the wide range of pos- 
sible study designs and the intricacies of design 
of quesrionnaires and subsequent coding and 
transformation of variables; A) environmental 
and biomarker sample-collection procedures, 
chain-of-custody and sample processing and 
storage, laboratory analyses, data analysis, and 
coding and c) imputation of missing vari- 
ables or laboratory nondetects. Although it is 
a standard practice to carefully document all 
of these details, there is currently no gener- 
ally agreed-on manner in which to upload 
.such data into an electronic database. There 
is a ride that people who were not involved in 
data coUeaJon can misunderstand these details 
and tbu-s obtain erroneous results. Some effort 
would be required to develop a standardized 
system for reporting this kind of information. 
For experimental animal studies, there should 


be parameters related to quality assessment 
(e.g., blinding of investigators, randomization, 
housing and care of animals). 

if the U.S. EPA chose this path, the first 
step might be to develop a framework simi- 
lar to ClinicalTrials.gov that would capture 
statistics and other parameters but would not 
necessarily require uploading raw data. With 
adequate funding, involvement of the NLM 
might provide more sophisticated informat- 
ics expertise to make the data more usable, 
and the NLM or the National Toxicology 
Program (NTP) could perhaps provide a 
“home” for the data. The U.S. EPA and other 
environmental agencies could also require (or 
request) that investigators register their studies 
with the database, and journals asuJd require 
registration as a condition of publication (as 
some journals currently do for results of clini- 
cal trials) or suggest that it be done. Given 
resource limitations, especially for investigators 
in developing countries, this step might be 
{fifficuJt for many investigators compared with 
researchers who perform clinical trials. 

A .system char provides raw data might 
be possible if the U.S. EPA could pilot the 
development of a system that could handle 
raw data using data already in its possession 
[e.g., re.sults of its intramural research, results 
of U.S. EPA-funded extramural research 
(where available), and any raw data that it 
has requested from investigators in support 
of risk-assessment activities}. Other federal 
agencies, such as the NTP and the National 
Institute for Occupational Safety and Health 
could contribute as well. The NTP already 
publishes ail of Its data and methods in its 
tepurts; however, it does not publish raw data 
or studies with nonstandard protocok online. 
In any case, busy investigators may oppose rhis 
not only because of the effort and resources 
required but also because they would be relin- 
quishing exclusive access to tlveir own raw data 
(and therefore the risk of being “scooped”) for 
the possibility of future requests for reanalysis. 
Even in cases where investigators contemplate 
no further data analyses, they may have con- 
cerns about the effort to respond to questions 
about repeat analyses. In any case, additional 
resources would be required, and this is not 
a time of plenty for research in the United 
Stares or anywhere else. In short, as in all of 
life, there is no free lunch. We already have 
mechanisms for disclosure of data u.sed by die 
U.S. EPA in decision making and even for 
obtaining raw data. It is doubtful that we can 
afford the luxury of having this information 
available for release prior to any request, and it 
is uncertain who should be responsible for the 
cost and effort required to provide it. 

We conclude that, as is the ca.se for clinical 
triak, a registry for studies that could handle 
a wide variety of methodologies and methods 
of analysis and provide a more complete and 
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standardiiect presentation of statistical results 
and other parameters than is possible in the 
peer-reviewed literature would be a tremendous 
resource to society for increasing transparency 
and improving assessments of pesticides and 
chemicals. However, at present, there is no 
evidence that there is a net social benefit to 
requiring collection of and access to, raw data 
for all studies utilized by the U.S. EPA prior 
to requests for such data from interested 
parties. As a first step the U.S. EPA, NTP, 
and N1.M should begin to generate discussions 
among agencies and with interested outside 
parties, including academic researchers and the 
regulated industty, on the possible creation of 
a reporting S)'Stem for environmental health 
studies of chemicals and pesticides that would 
systematically collect results and data about 
studies — but not raw data. 
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Access to Chemical Data Used in 
Regulatory Decision Making 

htip://c/x.cioi.oig/lO. 1289/ehp. 1206438 

It is dear from our commentary (Goldman 
and Silbergeld 2013), that we disagree with 
Lutter et al. (2013) about whether the pub- 
lic disclosure of all raw data used by the 
U.S. Environmental Protection Agency (EPA) 
for making regulatory decisions for chemicals 
is necessary to ensure the scientific ba.sis for 
such decisions, and about the extent to which 
preemptive disclosure (prior to any request) 
is practical. However, the most important 
disagreement between us is the basis asserted 
by Lutter ec al. in their commentary for this 
change in policy. Lutrer ct al. argued that it 
is necessary for the U.S. EPA — and anyone 
else who desires to do so — to reanalyze all 
data used in their assessments in order to 
“replicate” the findings and conclusions of 
the original investigators. 


Lutter et al. (2013) repeatedly used the 
terms “replicability” and “replication” as 
synonymous with an “independent analy- 
sis” of raw data from an existing study. 
Replication in science is quite diHercnt; it 
involves performance of an independent study 
with the same hypothesis and then testing 
the extent to which this independent study 
reaches the same conclusions. Recalculation 
of study statistics or other reanalysis of an 
existing study data set is not a replication. 
Designing and conducting a rcpltcation study 
does nor require access to raw data from the 
original study; this would abrogate the con- 
cept of independence. Moreover, an indepen- 
dent study will by definition utilize different 
sets of animal models or human populations, 
and as a consequence may employ different 
statistical techniques. 

'ihdr second argument is that dtsdosure 
of raw data will assist in identifying sources 
of scientific bias. Wc consider this unlikely 
bcc3u.se (he most important sources of bias 
are usually rdared to problems in study design 
or lirmoucns of the data collected. This is not 
idcnrifiaMe through data recalculation; how- 
ewT. this type of bias can usually be idenrified 
in the text of the ordinal study publication. 

Liimr ct al. (2013) noted (correaly) that 
applicants to the U.S. EPA for pesdeide regis- 
trations must provide raw data from regula- 
tory ceuing as parr of the package submitted 
to the F.P.V This is a very special case, in 
chat thae snadics are neither peer reviewed rK>r 
accevJA- to the public because of the ptotec- 
don sought by industry and extended by law 
for coniUenclal business information (CBI). 
The assumption of bias related to these stud- 
ies is rtot unreasonable, given that th« 7 ' arc 
condbciod by or on behalf of commercial enti- 
ties seeking to obtain pesticide registration. 
These tfiidiri ore rarely published in the scien- 
dfic litcranirc or in any way subject to inde- 
pendent peer review other than review by the 
U.S. EP.^. Many scientists and public policy 
ptaednonen consider the CBI cloak as a major 
impediment to transparency and confidence, 
industry could demonstrate their commitment 
to transparency by declining this protection, 
thereby irrcrcasing the confidence of dL 

Finally. Luncr ct aL (2013) attempted to 
support thdr proposal by claiming that jour- 
nals [Nature and die Proceedings of the National 
Academy of Sciences of the United States 
(/’AM5)] and an expert body (the Bipartisan 
Policy Center) agree with them. However, 
these bodies have neither supported the con- 
cept of requiring that aB raw data be reported 
to the the U.S. EPA nor that the U.S. EPA 
carry out its own independent recalculation. 
Rather, Nature and PNAS require authors to 
agree to make data sets (as well as macctials 
and ptotocols) available to editors, and to oth- 
ers, upon request (Nature Publishing Group 


2012; PNAS 2012). One of us (L.R.G.) was 
a member of the Science for Policy Project; its 
final report (Bipartisan Policy Center 2009) 
also recommended this practice. Many jour- 
nals require data, such as DNA and protein 
sequences, macromolecular structures, micro- 
array data, and crystallographic data, to be 
made available on publicly accessible data- 
bases, but most of these are not “raw data” in 
the sense that Lutter et al. proposed. Nature 
also recommends that authors submit clinical 
trials data to external clinical trials databases 
(Nature Publishing Group 2012). 

In summary, we disagree with the argu- 
ment that raw data from every study i^ed by 
the U.S. EPA to support a regulatory assess- 
ment should be made available to the agency 
and to the public. This proposal does not 
serve the purpose of “replication” or identi- 
fication of bias, as asserted by Lutter et al. 
(2013). In practice, it may generate obstacles 
to good science and discourage researchers 
from .studying issues of importance in 
environmental health. This proposal would 
also limit the U.S. EPA from using the results 
of research published in the peer-reviewed 
scientific literature by placing studies off- 
limits if the authors did not submit raw 
data sets to the the U.S. EPA. 

Finally, there is no obvious need for these 
changes. When the U.S. EPA has determined 
a need to rcanalyrx data, the current regula- 
tory practice has not impeded such activities. 
Past history indicates chat difficult cases are 
rare and do not warrant an intrusive and 
burdensome new requirement for the auto- 
matic submission of data from all studies. 
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Access to Chemical Data: Lutter 
et at. Respond 

fmpJIdx. dot. org/i 0. 1289lehp. 1206438R 
We appreciate the attention paid by Goldman 
and Silbergeld (2013) to the issue of data dis- 
closure and agree chat there has been ‘‘increased 
demand for transparency and disclosure of the 
data used by the U.S. EPA [Environmental 
Protection Agency] to make evaluations that 
support regulatory decisions.” 

In their letter, Goldman and Silbergeld 
contend primarily that “replication” in science 
means to independently repeat a prior study 
to sec if the same results can be obtained. 
They suggest that public availability of the 
prior smdy’s data is unneccssatj’ because a sub- 
sequent study will generate its own data. In 
201 1, a special section of Science (Vol. 334, 
No. 6060) addressed replicability and repro- 
ducibility and made two general points. First, 
“replication,” as defined hy Goldman and 
Silbergeld, while perhaps the cornerstone of 
the scientific method, can be difficult in many 
senin^ because of the uniqueness of the pre- 
cise conditions surrounding field observations, 
the expense and time required to collea data 
(e.g., for longitudinal studies)., and ethical con- 
straints (c.g., Jasny et al. 20il). Second, in 
those cases where conduct of a second experi- 
ment may be impossible or infeasible, review 
and teanalysis of the first study’s data is still 
a meaningful step along the “reproducibility 
spectrum,” assists in understanding die dificr- 
cnces between a>mpeting analyses, and “may 
be sufficient to verify the qiiality of the scien- 
tific claims” (Feng 2011; see also loannides 
and Khoury 2011; Santer et al. 2011). 

Other empirical work also supports the 
view that data avallahility promotes repro- 
ducibility. In empirical economics, a disci- 
pline that use.s large-scale statisticai models 
broadly similar to those of epidemiologists, a 
famous study of replication of peer-reviewed 
research su^esred that inadvertent errors may 
be “commonplace rather than rare occur- 
rences” (Dewald et al. 1986). The American 
Economic Review (AER 201.3) subsequently 
adopted a policy “to publish papers only if 
the data used in the analysis arc clearly and 
precisely documented and are readily available 
to any researcher for purposes of replication.” 
Further, the A.ER conducted a recent evalu- 
ation of its policy and reported that about 
80% of 39 sampled papers met the spirit of 
the data availability policy (Glandon 2010). 
imponandy, independent efforts at replication 


of 9 selected papers found no serious errors 
(almost exact repllcarion for 5 studies and 
“several small dlscr^ancl^ ... immaterial to 
the conclusions” for another 4.) This result 
represents a marked improvement relative to 
the results of the oripnal 1986 study of rcpli- 
cition. The difference is presumably attribut- 
able, at least in pan, to the difference in care 
and quality of wt»k assodared with the AERs 
current policy of data availability. Although 
analytic methods underlying papers published 
in the AER arc different from those used in 
chemical evaluation, the experience of the 
AER su^^ that there is merit in promoring 
data availability for the purpose of improving 
the reliabLlity of the results of published, peer- 
reviewed scientific papers, at least in disdpiines 
that use complex staiisrical models. 

Finally, wc, like Goldman and Silbeigeld, 
"disagree with the argumenr that raw data 
from every study used by the U.S. EPA to 
support a regulatory assessment should be 
made available to the ^cncy and to the pub- 
lic.” Unlike Goldman and Silbergeld, wc 
recommend that the U.S. EPA. when it U-scs 
results of a published study in a regulatory 
assessment, ask the authors for underlying 
daca (Lutter et al. 2013). If the U.S. EPA does 
not receive such data, it should explain how it 
used the study results in iighi of the fea thae 
data sufficient to assess reproducibiliry was 
not forthcoming. We beiio-e our approach 
would fecilicace and not obamjcr good science 
and chat it would not dbeourafir naearcbers 
from studying issues of imporam m environ- 
mental health. Moreos'er. it would not, as 
Goldman and Silbergeld sue, 
limit the U..S. EPA from usin(t (he results of 
research published in the peer-iesiroed K«en- 
tific tiieraiure by placing sntdio off-limits if 
(he authors did not submit ta« d«u seo to the 
U.S. EPA 

RL.. an independeni (msuixani, cmsuln for 
CropLife America (CLA) anJ rrrrixfJ financial 
support from the CIji to moderate a forum 
and serve as principal author of this letter. C.B. 
consults for Dow AgroSciences LLC, an R&D- 
based agrochemical producer, registrant, and 
marketer. C.J.B. received CLA funding to review 
and analyze scientific literature on data quality. 
J.W.C. has previously received funding from the 
American Chemistry Council to author work 
on the quality of scientific research evaluating 
chemicaL D.E. consults for a tsariety of pesticide 
manujacturers and for the CLA. A.F. has con- 
sulted with nonprofit organizations funded by 
the CLA about pesticide issues. 
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Chairman SCHWEIKERT. Thank you, Dr. Silhergeld. 

Our last witness today is Mr. Raymond Keating, Chief Economist 
at the Small Business & Entrepreneurship Council. Mr. Keating 
has expertise on a wide range of issues affecting the entrepre- 
neurial sector of the U.S. economy. He has written eight books, 
hundreds of articles, and writes for the Small Business & Entre- 
preneurship Council and the Center for Regulatory Solutions’ on- 
line publication. Mr. Keating is also an Adjunct Professor at the 
Business School of Dowling College. He received his master’s in ec- 
onomics from New York University and an MBA in banking and 
finance from — is it Hofstra? 

Mr. Keating. Hofstra. 

Chairman ScHWEiKERT. Hofstra University. Mr. Keating, five 
minutes. 

TESTIMONY OF MR. RAYMOND KEATING, CHIEF ECONOMIST, 
SMALL BUSINESS & ENTREPRENEURS HI P COUNCIL 

Mr. Keating. Thank you, Mr. Chairman and Members of the 
Committee, for holding this important hearing today. 

As you mentioned, I am Chief Economist with the Small Busi- 
ness & Entrepreneurship Council and I noticed Representative 
Johnson left but I wanted to thank her for upgrading me to a doc- 
tor. I tried to get away with that because I have two master’s de- 
grees, but nobody really lets me, so I have to thank her when I get 
a chance. 

I am going to take a little different tack from my colleagues and 
look at this issue from the small business perspective and also from 
the public’s point of view of the regulatory process based on a sur- 
vey that our group did recently. So just a few points that I want 
to highlight from my written testimony, number one, you know, 
just to kind of — from a small business perspective, the costs of reg- 
ulation are very real and significant facts of economic life. Econom- 
ics 101 tells us that we should expect — what we should expect from 
increased regulation: higher costs for businesses and consumers, 
reduced market exchanges, and expanded political control, re- 
sources allocated based on political decisions and influences rather 
than via competition and consumer sovereignty, and that all wind 
up in the end diminishing economic growth. 

Number two, from a small business perspective, the SBA’s Office 
of Advocacy has done a study. Several times I believe they have — 
I think they have done it three times. They have updated it a cou- 
ple of times. Just looking at the costs of regulation, the costs of 
complying with regulation with an eye toward small business, 
those — just to throw out a few of those numbers, when you look at 
firms with less than 20 employees on a per-employee basis, the cost 
of complying with federal regulations are 42 percent higher than 
firms with 20 to 499 employees and 36 percent higher than firms 
with 500 or more employees. On the environmental front in terms 
of environmental regulations, those disparities are even much, 
much higher. 

So the issue of transparency on the science being used to support 
regulation is not, you know, an esoteric academic or political point. 
It is very — has very real consequences in terms of the costs im- 
posed on small businesses. And small business owners really want 
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to know. They need to know what regulations quite frankly are le- 
gitimate and which ones that they are dealing with may not be so. 

You know, there is a lot going on at the EPA in terms of green- 
house gas regulations and there is more coming. When you look at 
the industries that are going to be directly affected and are directly 
affected, again, the majority of those businesses — the vast majority 
are small firms. When you look at manufacturing firms, manufac- 
turing employer firms with less than 20 workers, that is 76 percent 
of those businesses. So they are small businesses. When you look 
at mining, quarry, oil and gas extraction, 85 percent of employer 
firms have less than 20 workers. So this is a very real issue for 
small businesses across the board. 

Now, the poll that I want to mention we released it last month. 
It was a poll of American adults under the Center for Regulatory 
Solutions, our new organization, our new group if you will. And it 
was interesting what the public had to say on both the process and 
the cost of the effects of regulation. On the process, three numbers 
real quick: 68 percent said that government regulations on busi- 
ness are created by out-of-touch people who are trying to push a 
political agenda, 72 percent said that government regulations are 
created in a closed, secretive process, 64 percent said that govern- 
ment regulation on business was created in a way that does not 
consider the real-world impact. So that is the public view of the 
regulatory process. 

In terms of the effects, 53 percent agree that there are too many 
regulations on business, 61 percent believe that regulations on 
business are likely to do more harm to the economy by interfering 
with the market, preventing businesses from growing and hiring 
new employees and increasing prices for consumers. And small 
business owners would most assuredly agree with those assess- 
ments. 

One other one, you know, there is a whole host and I will be 
happy to get you the results of these — this survey, but 70 percent 
of Americans said that regulations, they hurt the economy, 66 per- 
cent said they mostly hurt entrepreneurs and small businesses. 
Hurt consumers, 63 percent, mostly hurt American workers, 66 
percent. You get the idea. The numbers are overwhelming in terms 
of how we are viewing — how the American public views this proc- 
ess. 

When you look at the economics of regulation, the impact of reg- 
ulatory costs on small businesses, the views of the public on the 
regulatory process really should push government officials to be 
transparent in all aspects of regulation, including how regulations 
are created, the scientific reasons for regulation, the true cost of 
regulations. And it matters — ^you know, it is — you don’t want to 
have a situation where certain agencies or certain political points 
of view or certain political members are deciding who gets access 
and who doesn’t. 

So I think when you look at the Secret Science and Reform Act, 
I think everybody in Congress on both sides of the aisle should be 
able to support it. 

Thank you. 

[The prepared statement of Mr. Keating follows:] 
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Chairman Schweikeit, thank you for hosting this important hearing today on the need for making 
the science EPA uses to justify regulatory costs on businesses and the economy, including small 
enterprises and ultimately consumers, more open and accessible to the public. This will help in 
holding EPA more accountable to stakeholders of all kinds — most especially the workers who 
ultimately have to comply with new regulations. Those on both sides of the political aisle and 
the regulatory debate should support transparency of the underlying data, science and analysis 
used to justify government regulation. It’s certainly an imperative for small businesses, given 
that regulatory costs disproportionately harm smaller firms. 

The Small Business & Entrepreneurship Council (SBE Council) is pleased to submit this 
testimony on behalf of our Center for Regulatory Solutions. 

My name is Raymond Keating, and I am the chief economist for SBE Council, as well as serving 
as an adjunct professor in the Townsend Business School at Dowling College where I teach a 
variety of courses in the MBA program; a weekly newspaper columnist for Long Island Business 
News; and author of several books, with the latest being Unleashing Small Business Through IP: 
Protecting Intellectual Property, Driving Entrepreneurship. 

SBE Council is a nonpartisan, nonprofit advocacy, research and training organization dedicated 
to protecting small business and promoting entrepreneurship. With nearly 100,000 members and 
250,000 small business activists nationwide, SBE Council is engaged at the local, state, federal 
and international levels where wc collaborate with elected officials, policy experts and business 
leaders on initiatives and policies that enhance competitiveness and improve the environment for 
business start-up and growth. The Center for Regulatory Solutions is a project of SBE Council. 

The Costs of RegaiatioD 

In 1986, President Ronald Reagan declared, “Government’s view of the economy could be 
summed up in a few short phrases: If it moves, tax it. If it keeps moving, regulate it. And if it 
stops moving, subsidize it.” President Reagan had a way with words - in this case, an ability to 
drive home an economic fact of life about the serious costs of various government actions, 
including regulation - in an amusing way. 

The costs of regulations are real and significant facts of economic life about which small 
business are too often painfully aware. Economics 101 tells us what to expect from increased 
regulation - that is, higher costs for businesses and consumers, reduced market exchanges and 
expanded political control, resources allocated based on political dictates and influences (such as 
rent seeking) rather than via competition and consumer sovereignty, and therefore, diminished 
economic growth. 

For example, economists John Dawson at Appalachian State University and John Seater at North 
Carolina State University recently looked at the impact of federal regulation on economic growth 
(“Federal Regulation and Aggregate Economic Growth,” January 2013). Their findings were 
striking. They reported: 
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“Regulation’s overall effect on output’s growth rate is negative and substantial. 

Federal regulations added over the past fifty years have reduced real output 
growth by about two percentage points on average over the period 1949-2005, 

That reduction in the growth rate has led to an accumulated reduction in GDP of 
about $38.8 trillion as of the end of 201 1. That is, GDP at the end of 201 1 would 
have been $53,9 trillion instead of $15.1 trillion if regulation had remained at its 
1 949 level.” The authors added: “Our results are qualitatively consistent with 
those obtained ifom studies using the various cross-country and panel data sets on 
regulation. Quantitatively, our estimated impact of regulation on aggregate output, 
large as it is, is similar to or lower than the micro-level impacts estimated in the 
cross-country and panel data studies. The cross-country and panel data are 
constructed very differently from our data, covering a subset of total regulations 
but over an array of countries. It thus seems that regulation has strong and robust 
negative effects on aggregate output.” 

Another look at the state and costs of federal regulations was provided in the twentieth 
anniversary edition of “Ten Thousand Commandments: An Annual Snapshot of the Federal 
Regulatory State,” published in 2013. The author, Clyde Wayne Crews Jr., reported, “For the 
first time in history, the estimated cost of regulation exceeds half the level of the federal budget 
itself.” It cost Americans an estimated $1,806 trillion to comply with federal regulations in 2012. 
Combine federal spending with these estimated regulatory costs, and “the federal government’s 
share of the entire economy now reaches 34.4 pcrccnL" That’s a serious drain and drag on the 
private sector. 

How do recent regulatory costs compare to the past? According to Crews, the Federal Register 
serves as government’s “depository of all proposed and final federal rules and regulations,” and 
its number of pages has long served as a rough measure of the scope of federal regulation. Crews 
pointed out, “Three of the four all-time high counts have occurred during the Obama 
administration.” 

The Obama years have been particularly troublesome in terms of “economically significant” 
rules, that is, those that impose an annual cost on the economy of at least $100 million. As noted 
in the report, “when it comes to economically significant rules at the completed and active stage 
. . . the current administration is in a class by itself when one looks at the year-end flow.” 
Economically significant rules have been higher in each year during the Obama administration 
versus each year during the Bush administration. In fact, the highest annual level under Obama 
was 24 percent higher than the peak level under Bush. 

Crews also broke out regulations by departments of the federal government. He found that the 
departments of the Treasury, Commerce, the Interior, Agriculture, and Transportation accounted 
for 1,730 rules, or 42.6 percent of all rules in the agenda pipeline. The Environmental Protection 
Agency (EPA) came in sixth, and once the EPA’s 223 rules are factored in, those six 
departments tallied up to 1,953 rules, or 48 percent of all rules in the pipeline. 

The Small Business Administration’s Office of Advocacy periodically estimates regulatory 
costs, obviously with an eye towards the burdens imposed on smaller businesses. In September 
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2010, the Office of Advocacy published an updated study estimating the costs of complying with 
federal regulations. The study - “The Impact of Regulatory Costs on Small Firms” by Nicole V. 
Crain and W. Mark Crain from Lafayette College - provided details regarding the burdens of 
federal regulatory costs. For example: 

• The annual cost of federal regulations registered $1.75 trillion in 2008. 

• For firms with less 20 employees, the per-employee cost registered $10,585, which was 42% 
higher than the $7,454 per employee cost for firms with 20-499 employees, and 36% higher than 
the $7,755 for firms with 500 or more employees. 

• On the environmental front, per employee regulatory costs for firms with less than 20 
employees came in at $4,101, which topped the $1,294 cost for firms with 20-499 employees by 
217% and the $883 cost for businesses with 500 or more workers by 364%. 

• Small manufacturers get hit particularly hard. Per employee regulatory costs for manufacturers 
with fewer than 20 employees came in at $28,3 1 6, which was 1 1 0% higher than the S 1 3,504 for 
manufacturers with 20-499 employees and 125% more than the $12,586 burden on companies 
with 500 or more employees. Again, serious cost differentials came in the area of environmental 
regulation, where per employee costs for manufacturers with fewer than 20 employees came in at 
$22,594, which topped the $7,131 for firms with 20-499 employees by 217% and exceeded the 
$4,865 for firms with 500 or more workers by 364%. 

Of course, it needs to be pointed out that small and mid-size businesses - that is, those with less 
than 500 workers - are central to economic growth and job creation. As the SBA’s Office of 
Advocacy has summed up (“Frequently Asked Questions About Small Business,” September 
2012), small businesses account for 46 percent of private-sector output, and 98 percent of firms 
exporting goods. As for jobs: “Small firms accounted for 64 percent of the net new jobs created 
between 1993 and 2011 (or 1 1.8 million of the 18.5 million net new jobs). Since the latest 
recession, from mid-2009 to 201 1, small firms, led by the larger ones in the category (20-499 
employees), accounted for 67 percent of the net new jobs.” 

Given the formidable costs of regulation, including on small businesses, the need for the 
regulatory process to be transparent should be supported, again, by both sides of the political 
aisle, and by both sides of the regulatory debate. 

The issue of transparency regarding the science being used to support regulation is not some 
esoteric academic or political point. It has very real world consequences in terms of costs 
imposed on small businesses, and the resulting fallout for economic growth and job creation. 

Secret Process: Social Cost of Carbon 

One of the most glaring cases of regulatory secrecy relates to President Obama’s climate agenda. 
In a secretive process, several agencies dubbed the “Interagency Working Group” established a 
highly speculative cost estimate called the “social cost of carbon” (SCC) to measure the benefits 
of reducing carbon emissions. 
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As described in a May 2013 document from the White House, “The SCC is an estimate of the 
monetized damages associated with an incremental increase in carbon emissions in a given year. 
It is intended to include (but is not limited to) changes in net agricultural productivity, human 
health, property damages from increased flood risk, and the value of ecosystem services due to 
climate change.” 

From 2010 to 2013, the administration’s SCC estimates, according to various models, jumped 
markedly, with the four SCC estimates (priced in 2007 dollars per metric ton of C02 at various 
discount rates) for 2020 moving from $7, $26, $42 and $81 in the 2010 analysis to $12, $43, $65, 
and $129 in the latest assessment. That’s quite a leap higher in a short period of time. 

In a September 2013 letter, a coalition of business groups pointed out, rightly, that the 
Administration’s SCC estimates “are the product of an opaque process, are fraught with 
uncertainties, and any pretensions to their supposed accuracy (and therefore usefulness in policy- 
making) are unsupportable.” To date, the Administration has not been forthcoming about who 
specifically participated in the process and whether the IWG adhered to federal guidelines in 
crafting the SCC estimate. The Administration’s stonewalling prompted two members of 
Congress to seek a Government Accountability Office investigation to uncover key information 
related to the IWG process. To be sure, this is not, as the President promised, “the most 
Iransptu'ent Administration in history.” 

It turns out that the IWG was secretive because they had something to hide, including the fact 
that models the group used to calculate the SCC estimate are effectively worthless. This is not 
just my opinion, but that of Professor Robert Pindyck of the Massachusetts Institute of 
Technology, an expert on these specific models. In a September 3, 2013, Wall Street Journal 
article, it was reported: 

"Robert Pindyck, an economics professor at the Massachusetts Institute of 
Technology, slams the models in a coming paper to be published by the National 
Bureau of Economic Research, saying they use essentially arbitrary inputs and 
give a misplaced illusion of scientific certainty. Though his work has given 
ammunition to skeptics of global-warming science, Mr. Pindyck said his point is 
really about the difficulty of modeling possible catastrophic impacts of climate 
change. ‘We know there's a social cost of carbon, and we know it's above $0,’ he 
said. ‘If anything, the cost of carbon could be higher’ than the administration's 
models suggest.” 

Pindyck’s point, again, is that this kind of modeling is effectively bogus. It offers nothing 
scientific - no matter which side of the debate you happen to be on. 

This issue of the “social cost of carbon” is critical, given that EPA’s greenhouse gas regime will 
eventually cover the entire economy, covering a wide array of industries, including pulp and 
paper, cement, oil and gas, chemicals manufacturing, mining, and many more. 
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As is the case throughout the U.S. economy, industry after industry in these cases 
overwhelmingly is about small and midsize firms. For example, 76 percent of U.S. 
manufacturing employer firms had less than 20 workers in 2010 (latest Census Bureau data), and 
98.6 percent had less than 500 workers. Or, consider that within the mining, quarry, and oil and 
gas extraction sector, 85 percent of employer firms had less than 20 workers, and 98.4 percent 
less than 500 employees. 

When it comes to costly regulations being imposed based in part on faulty, speculative “social 
costs of carbon” models, small businesses will bear a heavy burden, thereby limiting investment, 
economic growth, job creation, and competitiveness in the global economy. 

Hiding Science at the EPA 

In August 2013, the House Science, Space, and Technology Committee Chairman Lamar Smith 
issued a subpoena to the EPA for the release of the science used as the basis for costly air 
regulations. 

As noted in the accompanying release: “Over the past two years, the Committee has repeatedly 
requested the data the agency uses to justify virtually every Clean Air Act regulation proposed 
and finalized by tbe Obama administration. This was the first congressional subpoena the 
Science Committee has issued in 2 1 years.” 

It was also reported: “The two data sets in question are used to justify major costly new air 
regulations. As one example, by its own estimates the EPA’s proposed limits on ozone will cost 
taxpayers $90 billion per year, making it the most costly regulation the federal government has 
ever issued. Some of the data in question is up to 30-years-old.” 

Getting the underlying data in question is critieal to ensuring that EPA’s upcoming ozone 
rulemaking — not to mention the entire suite of ambient air quality standards EPA will establish 
in the coming years — is based on the most rigorous science, and that the public has an objective, 
accurate assessment of the costs and benefits of this rulemaking. 

The U.S. has made enormous progress in cleaning the air over the last 40 years, so much so that 
we now are talking about reducing very small increments of pollution. Achieving those tiny 
reductions will no doubt be very costly — as EPA itself admitted when it released its cost analysis 
for ozone in 2010. The question is: will they be worth it? We won’t know that unless we have 
the scientific data in front of us, unless scientists from all over the country can attempt to 
replicate it and determine its validity. Without that, EPA is hiding the ball, and imposing costs 
without truly knowing what the benefits are. Workers at small firms have the right to know what 
they’re paying for and complying with. Hopefully, it won’t cost them their jobs. 

The Incentives to Regulate 

Regulation is economically dangerous because the costs arc hidden from the eyes of the average 
person. People can see the bottom line on taxes, such as smaller take-home pay when income 
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taxes climb or increased costs at the cash register when sales taxes rise. But the costs of 
regulation, while just as real and significant as taxes, are not so clear. 

Therefore, politicians like to take credit for various initiatives through regulation, while leaving 
it to business owners and managers to wrestle with the commensurate costs, including, for 
example, having to raise prices, reduce payroll, rein in or eliminate expansion plans, or even 
close up shop. 

For good measure, keep in mind that both the Congress and the president not only have 
incentives to issue regulations, but they also have every incentive to pass off the actual 
rulemaking to unelected bureaucrats. This is a toxic recipe for rising regulatory burdens, reduced 
accountability, and yes, less transparency. 

This incentive stmeture is another reason why there should be no questions, or secrecy, regarding 
the data and the science that supposedly justify regulations. 

The Public View on Regulation 

Interestingly, a poll released in January 2014 by the Small Business & Entrepreneurship 
Council’s Center for Regulatory Solutions revealed notable concerns among American adults 
regarding government regulation. For example, regarding the process of regulation: 

• 68% said that government regulations on business are “created by out-of-touch people who are 
trying to push a political agenda.” 

• 72% said that government regulations are “created in a closed, secretive process.” 

• 64% said that government regulations on business arc “created in a way that does not consider 
their real-world impact.” 

• 61% agreed that government regulations are “administered ineffectively without rhyme or 
reason.” 

As for the effects of regulation, consider; 

• 53% agreed that there are too many regulations on business, while 19% said there are not 
enough and 25% said about the right amount. 

• 61% believe that government regulations on business are more likely to “harm the economy by 
interfering with the fi-ee market, preventing businesses from growing and hiring new employees, 
and increasing prices for consumers.” 

• As for the overall effect of government regulations on business, it was found that 70% of 
Americans said that regulation “mostly hurt” the “American economy,” 67% said they mostly 
hurt “America’s economic competitiveness with the rest of the world,” 66% said they mostly 
hurt “entrepreneurs and small business,” 63% said they mostly hurt consumers, 64% said 
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regulation mostly hurt employers, 66% agreed they mostly hurt “American workers,” and 58% 
said “American companies’ ability to innovate” was mostly hurt by regulation. 

These findings on the process and costs of regulation should further push government officials to 
be transparent in all aspects of regulation, including how regulations are created, the scientific 
basis for regulation, and the true costs of regulation. 

The Need for Transparency and Soundness When Regulating 

In President Obama’s Executive Order 13563 titled “Improving Regulation and Regulatory 
Review,” it was stated: 

Our regulatory system must protect public health, welfare, safety, and our 
environment while promoting economic growth, innovation, competitiveness, and 
job creation. It must be based on the best available science. It must allow for 
public participation and an open exchange of ideas. It must promote predictability 
and reduce uncertainty. It must identify and use the best, most innovative, and 
least burdensome tools for achieving regulatory ends. It must take into account 
benefits and costs, both quantitative and qualitative. It must ensure that 
regulations are accessible, consistent, written in plain language, and easy to 
understand. It must measure, and seek to improve, the actual results of regulatory 
requirements. 

That’s a sound declaration, and steps obviously need to be taken to shift away from the 
unfortunate realties of regulation, and closer to this statement. 

Clearly, the “Secret Science Reform Act of 2014” would be a straightforward, common-sense 
reform. As the bill requires, EPA “shall not propose, finalize, or disseminate a covered action 
unless all scientific and technical information relied on to support such covered action” is 
“specifically identified” and “publically available in a manner that is sufficient for independent 
analysis and substantial reproduction of research results.” 

That would be a valuable, simple reform measure that all in Congress and in the Obama 
Administration - based on the President’s own Executive Order - should agree on, and that the 
American public apparently would embrace given their views on regulatory process and costs. 
And I can assure you that the members of the Small Business & Entrepreneurship Council 
embrace this reform, as it would take the scientific information EPA regulators use and send it 
into the public realm, where it can be properly debated and analyzed. As EPA’s costly ozone 
rulemaking looms over our stagnant economy, it would be welcome indeed if EPA would be 
required to make public the all the data it intends to use. What do they have to hide? After all, 
wouldn’t EPA want to be absolutely certain that a potentially $90 billion rulemaking is worth it? 
I know small businesses and their workers certainly would. 

Thank you for this opportunity to address the Committee, and I will be glad to answer any 
questions. 
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Chairman SCHWEIKERT. Thank you, Mr. Keating. 

One of the joys of getting to sit in this chair, apparently I get 
to do the first question. 

Professor Graham, from what you have heard today and one of 
my premises is it goes far beyond just sort of us talking about the 
EPA so that the premise of sort of crowdsourcing of access to data 
and that the vetting of the confidence within — you know, the statis- 
tical confidence. Wouldn’t most of the — well, from the regulatory 
community, the research community, even sort of the armchair 
statistician, does that really hurt environmental science or research 
or would it in some ways make it more robust? 

Dr. Graham. I certainly don’t think it hurts it at all and there 
are lots of examples where it has helped with a more robust discus- 
sion. 

I do want to add though and underscore I think a useful distinc- 
tion that Dr. Silbergeld made between independent studies that 
draw their own data and whether they verify other studies versus 
just reanalysis of existing information, and I agree with her that 
the independent studies sometimes are much more powerful and 
important. You do need to have a lot of clarity and transparency 
about how the original study was designed and its protocol in order 
to do that properly. So a lot of the requirement that is in the bill 
is necessary to do good independent studies. But I also think rea- 
nalysis is oftentimes a very useful type of work that adds new in- 
sights. And my example of the automobile safety regulation is one 
of those. 

Chairman ScHWEiKERT. Well, and almost to that premise — and 
maybe Dr. Cox would be appropriate for this, when looking at oth- 
ers’ studies, the ability to take, you know, a study that may be a 
couple years old and take — ^you know, and stress it, see what is ac- 
tually happening with the tails, sometimes bounce it against a 
more current study, you know, particularly in sort of a peer-re- 
viewed world of — I — and, forgive me, but I don’t know how much 
peer review is reading of the article and seeing if the general sta- 
tistics work or actually sort of having the ability to look at the un- 
derlying data and bounce it against other studies that are around 
and say do I have a level of confidence? So there are two questions 
there. Tell me about how far into the raw data you think the peer- 
review world is going, particularly with government-funded studies, 
and how important it is to be able to constantly take studies and 
sort of bounce other models and other data against it. 

Dr. Cox. The current crisis in non-reproducibility of studies and 
in publication of results that turn out to be false is solidly ground- 
ed in the existing peer-review system. So when we look at papers 
like why most published research articles are wrong or if we look 
at last month’s editorial on reproducibility in science, what is being 
referred to specifically is peer-reviewed studies. Typically, peer re- 
viewers don’t have the time or the opportunity to dig deeply into 
the original data. That is not the purpose of peer review. Peer re- 
view does add value by saying whether a paper makes sense, 
whether there are obvious methodological flaws. That is about as 
far as it usually goes. And again, the very severe problems with 
trustworthiness of published results and the excess of false 
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positives in the environmental and medical literature, which has 
been well documented, are based on peer review. 

To your second question, having the opportunity to throw new 
models at old data I think is critical for making progress. For ex- 
ample, in this world of environmental health that we all care about 
today, the key question of do exposures cause health effects is one 
that requires methods that have not traditionally been used. It is 
a great opportunity but it requires access to data in order for the 
new methods and better methods to inform public policy. 

Chairman ScHWEiKERT. Dr. Graham, just because this is a level 
of personal curiosity, I am a little bit of a taleb fan, sort of the con- 
cept of if it is in the tail, you don’t dismiss it because the cata- 
strophic event can’t happen. Is making data more egalitarian, will 
that actually provide us the opportunity to realize there is some- 
thing, whether it be environmental or in my view of the world, you 
know, all sorts — to actually be able to identify those risk profiles? 

Dr. Graham. It is a good question. The commentary this morning 
about the Harvard and American Cancer Society procedure where 
they designate legitimate researchers and say that their data 
would be made available to legitimate researchers, I just want to 
make sure that everybody realizes — I say this as a former Harvard 
faculty member — this is not open access. This is not public access. 
So some people who may have some very good ideas and could do 
very good analysis may not look on the face of it like they are le- 
gitimate to Harvard or legitimate to the American Cancer Society. 

So, yes, it is very important that the principle of open access and 
public access provide everyone an opportunity to participate and 
they may find some results that are very unusual, that challenge 
conventional thinking, and they may never have been judged legiti- 
mate when they started that work so it is a very important advan- 
tage of true public access. 

Chairman SCHWEIKERT. All right. And thank you, Mr. Graham. 
I know I am over time but I have two others I just wanted to touch. 
And I don’t think I have ever sat on a hearing where, when you 
look at the CVs of all of you, to quote my little brother, you are 
all freaky smart. And, forgive me, is it Dr. Silverberg? 

Ms. SiLBERGELD. Silbergeld. 

Chairman SCHWEIKERT. I actually got up very early this morning 
and actually read your “Evidence-based Toxicology” article, and I 
have got to compliment you. When a novice like myself could follow 
it, read it, and actually understand it, you are a terrific writer. 

I did want — there was just one thing in your conclusions and I 
appreciate the concerns so now I sort of want to sort of make the 
concept sort of move forward sort of public data for public policy. 
In your first couple paragraphs of your conclusion you actually sort 
of talk about requiring a framework to accommodate data from nu- 
merous types and that that may be sort of like the direction 
where — I know you were speaking of toxicology research goes. Do 
you actually see this happening sort of in the toxicology world 
where more and more data is becoming more and more accessible 
and a variety of researchers are analyzing it and weighting it and 
stressing it? 

Ms. Silbergeld. Thank you. If I may first respond to your con- 
cept of crowdsourcing science, I am not sure that is such a great 
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idea and I think, with all due respect to my colleagues on this 
table, the record of the tobacco industry in — going beyond stressing 
data — I would say subjecting it to the Spanish Inquisition to twist 

it to say something that we now know it never said 

Chairman ScHWEiKERT. Well, but 

Ms. SiLBERGELD. — is something that is disturbing. 

Chairman ScHWEiKERT. But, Doctor, I appreciate that but 
crowdsourcing of the data is substantially a new phenomenon, not 
from 20 years ago, and the ability for me to have taken those data 
sets and said look what is happening — I am sorry. It is not ceteris 
paribus. You are talking two different time frames and two dif- 
ferent technologies. 

Ms. SiLBERGELD. No. 

Chairman SCHWEIKERT. Yes, you are. 

Ms. SiLBERGELD. With respect, sir, that is why I am worried 
about the notion newly introduced of crowdsourcing because we 

have experience 

Chairman SCHWEIKERT. Okay. 

Ms. SiLBERGELD. — of what has happened to access of data. 
Chairman SCHWEIKERT. Okay. 

Ms. SiLBERGELD. That is my comment. With respect to your 
question, sir, I — and with respect to the comments about peer re- 
view, as a journal editor I will take those blows 

Chairman SCHWEIKERT. Well, that 

Ms. SiLBERGELD. — and I understand them 

Chairman SCHWEIKERT. — actually wasn’t my question. 

Ms. SiLBERGELD. Yeah. 

Chairman SCHWEIKERT. My question was on your conclusion here 
where you talked about more accessibility to data. I think I am 
going to have to wait until the next round because I am now 3- 
1/2 minutes over time already. So let me turn to my Ranking Mem- 
ber. Maybe she can follow up where I was going. 

Ms. Bonamici. Thank you, Mr. Chairman. 

And before I begin my questions, I — two brief points. First, I 
don’t want the lack of absence of Members on my side to be indic- 
ative of a lack of interest in this issue. There are several hearings 
and markups happening simultaneously, so please don’t consider 
this as a lack of interest in the topic. 

Also, Mr. Chairman, we received and provided you with copies of 
letters from the American Lung Association, the American Thoracic 
Society, the Union of Concerned Scientists, the Center for Progres- 
sive Reform, and the Natural Resources Defense Council stating 
opposition to this bill, and I ask unanimous consent that they be 
submitted to the record. 

Chairman SCHWEIKERT. Any objections? So ordered. 

[The information appears in Appendix II] 

Ms. Bonamici. Thank you, Mr. Chairman. 

Dr. Silbergeld, under current law, because of the need to ensure 
that protected information like health records remains confidential, 
there are likely to be data that will not and should not become en- 
tirely available to the public. So should the EPA be precluded or 
any agency for that matter — ^but for the purposes of this hearing, 
should the EPA be precluded from considering studies that include 
health information if the studies are significant to determining the 
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appropriate course of action? And if not, then how can we inde- 
pendently validate and verify such studies? 

Ms. SiLBERGELD. Thank you for the question. I think we have a 
model to go forward here. Which is really the model of evidence- 
based decision-making that first began in medicine, extended to 
healthcare, and now I count myself as one of the leaders in moving 
it into the field of toxicology in which we take the broadest possible 
look through the available — publicly available data and publica- 
tions, including government reports, and attempt to synthesize 
those data using transparent and open processes of data access and 
evaluation. So I think we should always be committed to extending 
our view as wide as possible. Do we need to see raw data? I am 
not convinced that we do. I think in those instances where it has 
become of interest to do so, we have methods in place to accomplish 
that. 

And perhaps in response to the Chairman’s question — and I 
apologize for using up your valuable time, sir — in fact, journals are 
increasingly soliciting and accommodating the production of much 
more extensive data, for example, on statistical models and proto- 
cols through the use of appendices. This is a somewhat new process 
in publication in which my own journal and others now encourage 
authors to submit this type of information which is publicly acces- 
sible through linkages in the paper and in any repository that has 
that paper that amplifies the kinds of information I think some of 
my colleagues would find particularly interesting. 

I also want to defend my reviewers by saying that they do a very 
exhaustive job of reviewing. And I agree with Dr. Cox that in fact 
I think some of the lack of reproducibility has come about through 
some failures in peer review. 

Ms. Bonamici. Thank you. And I am going to follow up on that 
because you state that requiring — in your article you state that re- 
quiring the disclosure of the raw data from every study that the 
EPA uses to support its regulatory assessment could actually pre- 
clude the EPA from using relevant research if the journal authors 
don’t submit the raw data to the EPA. So could you talk a little 
bit — I know you touched on that, but could you talk about what 
would be the impact to the EPA of limiting the scope of those stud- 
ies if they cannot consider those as part of their scientific review? 

Ms. SiLBERGELD. Well, for example, unless you can actually re- 
solve the problems of protection of human subjects who appear in 
studies, this would then eliminate a great deal of the epidemiologic 
and clinical literature that could be available. And I know that we 
worry about this because in fact it was the tobacco industry who 
tried to bust open some repositories of confidential data that would 
have permitted identification of human subjects, and the rationale 
given by the industry was they wanted to interview some of those 
subjects and see if they gave the same answers that were reported 
in questionnaires used in studies. So this is a very disturbing as- 
pect to the certainly of protection of people’s autonomy, confiden- 
tiality, and the process of scientific research. 

Ms. Bonamici. Thank you. And I want — time for one question if 
I may, Mr. Chairman. I understand that the broad scientific com- 
munity is engaged with the Office of Science and Technology Pol- 
icy, or OSTP, and what representatives from academia, particularly 
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the American Association of Universities and the Association of 
Public and Land-Grant Universities, is calling a thoughtful bal- 
anced process to increase public access to the results of federally 
funded research. And that is a type of government-wide approach 
to improving public access to federally funded R&D that is pref- 
erable to the legislation that specifically targets the EPA, especially 
in light of the fact the EPA generally relies on studies that are not 
funded by the Agency but other agencies like NSF and NIH. 

So could you please comment on the need to take a government- 
wide approach to improving public access? And if you could com- 
pare the bill we are considering today with OSTP approach and 
which would strike a better balance between the need for trans- 
parency in the regulatory process, balancing the rights of private 
citizens, and the need for the EPA to use the best science available. 

Ms. SiLBERGELD. Thank you. I think a comprehensive and con- 
sistent approach is certainly to be advocated. The fact, for example, 
is that a great deal of the evidence that the EPA would find and 
has found useful was in fact funded by other agencies such as the 
National Institutes of Health, which has set up an effective and 
functioning program, speaks to the need for a consistent policy. 

But I would like to restate my very great concern that I have a 
very great interest as a scientist to be able to see industry data and 
I would like to see industry behind the proactive stance and record 
of the American Chemical Council and others in terms of opening 
the doors on their data. 

Ms. Bonamici. Thank you. And my time is expired but I would 
be interested in hearing from the other witnesses about whether 
they would support the disclosure of the industry data as well. 

Thank you, Mr. Chairman, and I yield back. 

Chairman SCHWEIKERT. Thank you. Ranking Member. 

And, Mr. Bridenstine. 

Mr. Bridenstine. Thank you, Mr. Chairman. And I would like 
to say I am proud to support your bill, Mr. Schweikert, H.R. 4012, 
the Secret Science Reform Act. 

I would like to bring up a slide if it is possible on the screen. 

[Slide] 
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Mr. Bridenstine. There it is. This is a quote from the President 
of the United States when he was campaigning. “So if somebody 
wants to build a coal-fired power plant, they can. It is just that it 
will bankrupt them.” That wasn’t based on any kind of scientific 
data. That is a quote of the President of the United States. It is 
a campaign promise. He interviewed with the San Francisco 
Chronicle November 7, 2008. This is a promise that he has followed 
through with. There are two coal-fired power plants in Oklahoma 
that are being shut down and the rates on my constituents are 
going up. The low estimate is 20 percent, some say as high as 40 
or 50 percent. Who does that affect? It affects the poor the most. 
This is President Obama’s war on the poor. They spend the biggest 
part of their budgets on utilities, and so they are the ones being 
affected the worst. 

And here is the thing: The transparency of the data that they are 
using to create the rules and regulations that are shuttering these 
coal-fired power plants are not transparent. Transparency and 
verifiability are fundamental principles of any scientific endeavor 
and should certainly be required in those supported by taxpayers. 
The EPA continues to violate these principles by preventing inde- 
pendent researchers from examining the data and replicating the 
studies which “support the Agency’s rulemaking.” My constituents 
in Oklahoma’s 1st District are paying the price, quite literally pay- 
ing the price, for the EPA’s politicizing the regulatory process and 
its secret science charade. 

Mr. Chairman, I would like to spend a minute explaining how 
the EPA’s secret science and groundless regulations will continue 
to needlessly harm my constituents and all Oklahomans. According 
to the Energy Information Administration (EIA), Oklahoma pro- 
duces four percent of the country’s crude oil and eight percent of 
its marketed natural gas. Oklahoma is leading the fracking revolu- 
tion to provide cheaper energy to all Americans. Oklahoma is the 
5th-largest shale gas-producing State, and 17 of the top 100 nat- 
ural gas fields are located in Oklahoma. Mr. Chairman, the EPA’s 
newest regulations on fossil fuel production and refining based on 
this secret science charade threaten my constituents with higher 
utility bills, less reliable electricity, and fewer jobs in Oklahoma’s 
booming oil and gas industry. 

Can we bring up the second slide there? 

[Slide] 
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Mr. Bridenstine. The latest — can we bring up the slide 2? 

The latest EIA analysis shows that coal- and natural gas-fired 
power plants provide about 90 percent, 90 percent of Oklahoma’s 
electricity generation. My district has two petroleum refineries in 
Tulsa and four natural gas power plants, one in Coweta, one in 
Tulsa, and two in Jenks. Thanks for bringing up the next slide 
there. 

[Slide] 
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Mr. Bridenstine. We are talking about 155,000 barrels of refin- 
ing capacity per day and 3,200 megawatts of production capacity. 
Can we bring up the next slide? 

[Slide] 
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Mr. Bridenstine. My 1st District constituents also use electricity 
produced by four coal-fired power plants, just — one just outside of 
my district in Oolagah, Muskogee, and Chouteau. That is another 
4,300 megawatts of production that the EPA wants to shut down 
based on scientific data and models that it will not publicly release. 

Can I get the next slide, please? 

[Slide] 
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Mr. Bridenstine. Mr. Chairman, the EPA has put a bull’s-eye 
on coal-fired power plants, a bull’s-eye. This is the regulatory agen- 
da here of the EPA. And once the EPA kills coal, let me be clear. 
They will come for natural gas. It is just a matter of time and we 
are already seeing that in the Obama Administration’s war on 
fracking. 

Can we bring up the next slide, please? 

[Slide] 
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Mr. Bridenstine. In 2012, the Institute for Energy Research es- 
timated that the EPA’s MACT and Cross State Air Pollution Rules 
will take 33 gigawatts of national electricity generation off-line, 33 
gigawatts, over twice the EPA’s modeling prediction. That is ten 
percent of our country’s coal capacity and also includes closing two 
natural gas-fired plants in Oklahoma, natural gas-fired plants in 
Oklahoma, one in Anadarko and one in Oklahoma City. The end 
result of the EPA’s regulatory horror show is higher prices for con- 
sumers and industry, less economic growth, and fewer jobs. 

Next slide, please. 

[Slide] 
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Mr. Bridenstine. According to EIA, Oklahomans pay about 20 
percent less for residential, commercial, and industrial electricity 
than the U.S. average. Killing coal and eventually natural gas pro- 
duction in my State will saddle my constituents already struggling 
in this sluggish Obama economy with higher utility prices on par 
with those experienced in heavily regulated States such as Cali- 
fornia. 

Next slide. 

[Slide] 
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Mr. Bridenstine. Kentucky has already seen massive layoffs in 
the coal industry. When EPA uses secret science to justify new reg- 
ulations, everyone is worse off except for EPA bureaucrats, as well 
as extreme environmentalists. Let me be clear. The EPA is now 
saying they are going to come after our woodstoves. Of course, a 
friend of mine, Thomas Massie from Kentucky, has gotten assur- 
ances from the EPA that if you like your woodstove, you can keep 
it. 

Let me ask one question for each of our panelists. Do any of you 
disagree with the principle that in the case of taxpayer-funded re- 
search or studies, the public should have access to the underlying 
data? Does anybody disagree with that? 

Yes, ma’am? 

Ms. SiLBERGELD. As stated in my testimony for the reasons 
given, I disagree with that, respectfully. 

Mr. Bridenstine. Noted. I yield back. 

Chairman ScHWEiKERT. Mr. Smith, Chairman of the full Com- 
mittee. 

Chairman Smith. Thank you, Mr. Chairman. 

Mr. Keating, let me direct my first question to you, and I appre- 
ciate the poll that you mentioned earlier. I had not seen those re- 
sults and they are very, very telling. And let me just highlight a 
couple of the results that you mentioned: 68 percent said that the 
government regulations on business are created by out-of-touch 
people who are trying to push a political agenda; 72 percent said 
that the government regulations are created in a closed, secretive 
process; 70 percent of Americans said the regulation mostly hurt 
the American economy. Those first two poll results that I men- 
tioned and that you cited as well go to the heart of why we need 
the legislation at hand. And I really don’t have a question for you 
other than to thank you for your testimony. If you want to add any- 
thing to your comments about why the legislation will have a posi- 
tive effect, you are welcome to. 

Mr. Keating. Well, no, and there is a reason why I wanted to 
bring that here. I mean you talk about people being smart. These 
folks to my right here are far smarter than I am, but I want to 
bring the small business perspective and this survey that we had. 
And it is — ^you know, when we talk about the egalitarian access to 
data and the politicization of this process, those are very real 
things that are having, as the Vice Chair pointed out, very real ef- 
fects on our economy in terms of jobs, small businesses, economic 
growth, all the way down the line. 

So I am a Madisonian on this. I love checks and balances. And 
the more checks and balances that we have, the more people we 
have looking at this, the more scientists, the more economists that 
we have looking at this, the better it is going to be for everybody 
because the ultimate point is not to politicize this. It is to get sound 
public policies. That is the bottom line. 

Chairman Smith. Thank you, Mr. Keating. 

Dr. Cox, as you know, some of the data that the EPA is relying 
upon to make rules and regulations is up to 30 years old. What are 
the disadvantages of using data that old when it comes to making 
decisions? 
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Dr. Cox. There are trends and changes in the statistics of public 
health. Old patterns are not predictive necessarily of current 
events. More to the point, what regulators want to do is to inter- 
vene in the world as it is now to change it and make it better. To 
do that they often need to understand how the world works now 
with the current mix of pollutants, with the current configuration 
of industry. Staring hard into the rearview mirror does not nec- 
essarily provide that information. So I think the problem with old 
data is in part that it is old. 

The other thing that I will note is that basically regulations on 
results that depend on data that are not currently available does 
indeed allow the Agency to use the best-available information but 
it also allows Agency to use the worst-available information. Both 
are part of the published record. Without current access to current 
data, it is extremely difficult to distinguish between the best and 
the worst results in the past literature. So rather than looking at 
dead results and dead literature, I think it is important to keep the 
data alive and to allow current questions to be informed by current 
analyses of current data. 

Chairman Smith. Okay. Thank you. Dr. Cox. 

And, Dr. Graham, a two-part question, I guess. One, what is the 
advantage of making this data publicly available, if you can go into 
some detail; and secondly, is it really difficult to make this data 
available to the various scientific researchers? 

Dr. Graham. I don’t think it is difficult. It is already done in 
most fields of science. The environmental health field is different 
because of its extensive reliance on some of the patient records we 
have been talking about, and therefore, there is going to be a meet- 
ing next month at the National Academy of Sciences where they 
discuss not whether these data should be released and shared but 
how to do so in a way that protects the privacy and confidentiality 
of those participants. And I have already encouraged the Com- 
mittee staff to attend and learn from that discussion. And I think 
what you will find is most of the scientific community with the ex- 
ception of the mental health field is already on board for this agen- 
da. And so we do have a ways to go persuading the environmental 
health research community to be involved with the open access 
issue. 

Two other points just to respond to previous questions, this ques- 
tion of industry data, the way I read the Secret Science Reform 
Act, it applies to industry data. This legislation requires industry 
data to meet the same standards as academic data or other forms 
of data. The only thing I can think of that people were referring 
to is confidential business information, which would be held in the 
exception that is in the draft legislation. But I can tell you from 
my experience at 0MB, 90, 95 percent of the industry information 
that they want to bring in is not covered by confidential business 
information. 

Chairman Smith. Okay. 

Dr. Graham. So this bill is going to cover industry data, as it 
should. Okay. Second point is should we have a government-wide 
approach, wait for all the agencies to agree that we should do open 
access or shall we do something that is focused on environmental 
health research? As I have mentioned, we already have open access 
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in most fields of science already. You need a targeted approach 
that goes right at the domain of science where this problem exists 
and you need a solution obviously to the patient record issue to as- 
sure confidentiality. Thank you very much. 

Chairman Smith. Thank you, Dr. Graham. It is very helpful. I 
yield back. 

Chairman SCHWEIKERT. Thank you, Mr. Chairman. 

Ms. Edwards. 

Ms. Edwards. Thank you very much, Mr. Chairman, and I apolo- 
gize for the shuffling, juggling a little markup in another Com- 
mittee this morning. 

Dr. Silbergeld, as you know, it has been important to protect per- 
sonal information of individuals participating in epidemiological 
studies and that there is a strict code of ethics among researchers 
as it relates to human subjects. As I understand it, researchers 
such as yourself are working on ways to disclose data from human 
studies so that individuals are protected but that others can use 
the data. However, in one of your articles you indicate that there 
are studies where the removal of all identifying data negates the 
scientific value of the data set. Can you elaborate on this statement 
in instances where the identifying data are necessary to the 
science? And also, can you discuss the importance of protecting the 
privacy of study participants and concerns that may arise if the po- 
tential study participants fear that their information will be ex- 
posed to the general public? 

Ms. Silbergeld. Thank you for that question. The clearest ex- 
ample and of greatest relevance to the subject of this Committee’s 
hearing has to do with studies in which we are interested in the 
location of the subjects. And we have very sophisticated methods 
of determining this through spatial statistics, the use of GPS, and 
other data. That data — those data have to be absolutely protected 
because if those data become available, people can pretty much 
identify who participates in the studies. And I have conducted sev- 
eral studies in which we collected those data. That information has 
to be completely de-identified, which then of course means that no 
one else can exactly take those data and carry out the same anal- 
yses. 

I just want to reiterate that that to me as a scientist reanalyzing 
data is an uninteresting approach to science and unlikely to ad- 
vance our confidence in the results of a specific study. What ad- 
vances my confidence and should advance the confidence of every- 
one in this room in the results from one study is really what Dr. 
Cox alluded to, is whether or not the study can be replicated inde- 
pendently, not whether obtaining the data from the first study can 
be reanalyzed. That is relatively, frankly, trivial. 

And in fact the data that are now being called upon by journals 
and the NIH will allow one to determine whether protocols are ap- 
propriate and statistical models were appropriately selected in 
studies designed to meet the requirements of hypothesis testing. I 
also agree that there should be adequate funding given to public 
and private sector entities so that we can have the updated sci- 
entific findings that we all agree are important to us. Thank you. 

Ms. Edwards. But would you say that there are instances where 
the identifying data set — that those are very limited instances of 
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research versus other kinds of research where you can have tons 
of data that it would he — you know, where it would he important 
to keep all of the data set available so that you could — I mean per- 
haps there could be an exclusion or some kind of a waiver or some- 
thing that isn’t the rule for more specialized identifying data sets 
where people’s personal information is identifiable? 

Ms. SiLBERGELD. I would have to say from my experience and as 
someone who routinely goes through the rigors of obtaining ap- 
proval from our Institutional Review Board, I cannot see that I 
have an ethical — I cannot ethically accept revealing information 
that would allow personal identification. 

I think that the most sophisticated approach to answering your 
questions and these difficult issues in addition to what Dr. Graham 
alluded to is really what is going on worldwide, probably first in 
the European Union because they are confronting it first, but we 
are looking at it now and that is looking at data from clinical trials. 
How do we get those data out there to restore the faith that has 
been pretty badly damaged by drug approval processes around the 
world and inadequate clinical trials but yet protect the identifica- 
tion of the persons who participate? And I think that is going to 
be a path forward that will be very informative for all of us. 

Ms. Edwards. Thank you. And I just have one question as my 
time is set to expire. Can you explain or describe rather what hap- 
pens when, as Dr. Graham has said, the scientific culture at EPA 
is fragile and still in an early stage of development and that the 
political, legal, and engineering cultures are stronger and more cer- 
tain than the culture of science and economics? You are the only 
scientist on this panel and someone who has worked on several ex- 
pert groups convened by EPA. Do you want to comment on that po- 
sition? 

Ms. SiLBERGELD. Thank you. I would like to defend the reputa- 
tion and international standing of the EPA’s scientific staff. They 
are widely recognized as among the best in the world. I even share 
the honor of a MacArthur Fellowship with a former EPA scientist. 
And while I am trained in engineering and I will accept his com- 
pliment, I think that the biological, ecological, and human health 
science at the EPA is extraordinary. 

Ms. Edwards. Great. Thank you very much. 

Thanks, Mr. Chairman. 

Chairman SCHWEIKERT. Thank you, Ms. Edwards. 

Dr. Broun. 

Mr. Broun. Mr. Chairman, before I start my questioning, I 
would like to take a point of personal privilege and just say that 
I am a physician; I am a scientist and we have seen throughout 
my medical career where there are reports and medical data that 
are not reproducible and where we have seen researchers promote 
an agenda and absolutely it is critical for us to have an open access 
to data so that we can reevaluate existing data of any study, as 
well as to reevaluate the findings by doing other studies that are 
likewise trying to study these same issues. 

Without having open access of data, it is absolutely impossible 
for a practicing physician such as myself to make a valid decision 
on drug use or whatever I am doing as a doctor to try to take care 
of my patient. So open access to all data is absolutely critical. And 
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also, in medical research we have avenues of protecting that per- 
sonal identifiable information and it is absolutely critical that we 
do so. 

So thank you, Mr. Chairman, for giving me access to a few min- 
utes to vent here a little bit. And I appreciate your bringing this 
bill forward because I think it is absolutely critical. 

And we have seen in Georgia, just like Mr. Bridenstine is talking 
about in Oklahoma, we have seen the EPA bring forth regulations 
as closing down 15 power plants in the State of Georgia. And we 
cannot get the data that they have brought forward to really evalu- 
ate why they are making the regulatory burden so heavy. And it 
is going to hurt poor people. This EPA has an agenda that is hurt- 
ing poor people and it is hurting senior citizens on a limited in- 
come. And having open access to the data is absolutely critical for 
us to be able to evaluate that. 

Now, having said that. Dr. Graham, in your view — I hope you 
will restore some time for me, Mr. Chairman. And, Dr. Graham, in 
your view does the Freedom of Information Act make this Secret 
Science Reform Act unnecessary? 

Dr. Graham. No, because oftentimes the government doesn’t 
have the possession of the data that is sought. 

Mr. Broun. Well, I agree with that. Dr. Graham, you mentioned 
that the EPA chemical assessments under IRIS program is an ex- 
ample of the nonregulatory EPA determinations that have signifi- 
cant impacts and should be subject to transparency requirements. 
The IRIS program has been criticized by the National Academy of 
Sciences and others. Do you believe greater data access would im- 
prove the program? 

Dr. Graham. Yes. 

Mr. Broun. Do you think the IRIS program could comply with 
the provisions of this bill, the Secret Science Reform Act? 

Dr. Graham. Yes. 

Mr. Broun. Dr. Graham, the office you previously oversaw at 
0MB, the Office of Management and Budget, produces a report of 
cost and benefits of regulations across the federal government. Ac- 
cording to the most recent draft of this report to Congress, the vast 
majority of the benefits for all regulations across the entire federal 
government, as high as 80 percent, are attributed to EPA Clean Air 
Act regulations and specifically reduced levels of fine particular 
matter. In your testimony you noted key uncertainties in EPA’s 
science on this question. Do you find these numbers credible? 

Dr. Graham. They have uncertainties, particularly given that the 
data can only be given to “legitimate researchers.” And the people 
who were sponsoring all this research get to decide who the legiti- 
mate researchers are. 

Mr. Broun. And it is just up to them. So it is secret about them 
deciding who 

Dr. Graham. It reminds me a little bit of the NFL concussion 
sort of debate we are having right now where the establishment 
community was saying there is no connection, there is no connec- 
tion, there is no connection and then sort of the people who were 
trying to bring other kinds of science involved. If you let the estab- 
lished community have complete control over who the legitimate re- 
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searchers are, you are not going to get unexpected and new in- 
sights. 

Mr. Broun. Mr. Keating, you mentioned forthcoming national 
EPA regulations on ozone, which could cost $90 billion per year. 
What kind of impact will this have on small businesses? 

Mr. Keating. Well, there is an assortment of impacts, first, the 
ones that you alluded to in terms of — think of a small business 
owner, small business as a consumer of electricity, power, so on 
and so on. So you are going to hit on that end in terms of the in- 
creased cost. But then as I mentioned in my testimony, all of these 
industries — and I love telling people this when I go out and speak 
to the public on all those issues. You know, if you look at the en- 
ergy industry, pick the pharmaceutical industry, all the ones that 
people think of as big oil and big pharma and big this and big that. 
When you actually dig into the numbers, the population is over- 
whelmingly small firms, less than 20 employees depending on the 
industry we are talking about, 70, 80 percent. So from a small busi- 
ness perspective, you are getting hit on both sides. You are getting 
hit as a consumer and you are getting hit as being part of the in- 
dustry, bottom line. 

Mr. Broun. Mr. Chairman, if I could, I just have another ques- 
tion for Mr. Keating to follow up on that. 

This President said that his energy policies will “necessarily sky- 
rocket the cost of energy.” This is what we are talking about as in- 
creasing the cost on small businesses. Who gets hurt most there? 
I believe it is poor people and senior citizens on limited income. 
Would you agree with that, Mr. Keating? 

Mr. Keating. I would agree with that but I certainly wouldn’t 
limit it there. When you are talking about an economy that is 
struggling to create jobs for as long as we have been struggling, 
who are the job creators? You know, again, when you look at the 
numbers, small- and medium-sized firms are creating roughly 2/3 
of the new jobs. There is a reason why we are suffering these days 
because — well, I know it could be a long list of things but I think 
they are all — they all go back to policy. And we look at regulatory 
issues, it creates an enormous amount of uncertainty for small 
businesses and it imposes an enormous amount of costs. Those 
are — you know, it is — again, the consumers — it is the small busi- 
nesses themselves but it goes well beyond that in terms of people 
that are just looking to get a job and get back into the labor force. 

Mr. Broun. Thank you, Mr. Keating. 

Mr. Chairman, I thank you for your indulgence. I am just very 
concerned about the attack upon energy, particularly coal-powered 
energy and fossil fuel energy that this Administration is utilizing 
the EPA to promote that attack, and it hurts poor people and sen- 
ior citizens I think the most but it also hurts job creators and hurts 
our economy. Thank you. I yield back. 

Chairman SCHWEIKERT. Thank you. Dr. Broun. 

Mr. Weber. 

Mr. Weber. Gosh, I was going to let Dr. Broun keep going. I 
wasn’t sure of my questions yet. I am just kidding. 

Chairman SCHWEIKERT. All right. Thank you, Mr. Weber. 

Mr. Weber. Next witness 

Mr. Broun. I tried to take your time. 
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Mr. Weber. Yeah. Yeah. 

I guess this is a question for all the panelists on the risk data, 
this data that is collected that we are discussing here today. And 
I will start with you, Dr. Graham. That is — ^by the EPA, for exam- 
ple, that is a public agency, right? 

Dr. Graham. Yes. 

Mr. Weber. Government-funded public agency, we would all 
agree with that? 

Dr. Graham. Yes. 

Mr. Weber. So I guess we would have to agree — anybody that 
disagrees with that? I guess we would have to agree that the 
money used to procure that data was public money. No? 

Dr. Graham. Well, the 

Mr. Weber. Dr. Silbergeld, you are saying no. 

Dr. Graham. It is a mixture. Some of the studies have a mixture 
of public and private money. 

Mr. Weber. All right. Do we question the motives of those who 
provide that private money? We seem to be questioning the motives 
of those who testify. 

Dr. Graham. At this hearing you are in good shape unless you 
are the tobacco industry. 

Mr. Weber. Well, that is what I am saying. Personally, I think 
they are blowing smoke but we seem to be questioning people who 
are involved in this — I am an air-conditioning contractor and — from 
the State of Texas and we want somebody on that licensing board 
that understands the industry or somehow is involved. And I un- 
derstand tobacco was a big lawsuit. 

The aim of science is to get to the truth and. Dr. Silbergeld, I 
was reading your personal statement here where you said, “there 
is an urgent and important need to adopt an evidence-based ap- 
proach to improve decision-making and increase public confidence 
in policy-making and enhancing the scientific basis of toxicology as 
well as its utility for other domains, including drug regulation,” on 
and on and on. And then you continue. You say, “through this proc- 
ess I think it is time — it is now appropriate to draw together a 
workshop of stakeholders.” Well, I would submit that we want peo- 
ple from industry, the energy industry, the coal industry, and we 
want the data to be accessible so that if — our aim is for the Amer- 
ican public’s health and its safety and we want — I mean is it not 
true that scientists want to get to the truth? Is that a truism? Or 
is it what Mark Twain said? All scientists are only sure of one 
thing and that is that every other scientist before them was wrong. 

Ms. Silbergeld. Well, actually, Mark Twain does express a cer- 
tain tenant in science, which is what we call the falsifiability prin- 
ciple advocated by philosophers, which is that we start by doubting 
our own hypotheses. 

Mr. Weber. Okay. Well, yeah. Well, that is right. And so I just 
want to point out that we want those industry ties and we want 
people to be able to do that. I just find it appalling that we seem 
to have a disagreement over people in industry have a legitimate 
reason for getting involved. We question their motive in so many 
cases. And I think that we have the underlying premise — and you 
tell me if I am wrong — that somehow government employees are to 
be trusted more. The EPA, Science Advisory Policy Committee, 
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whatever, are to be trusted more than those who have a vested in- 
terest in the very communities that they live in. And I want us to 
be open and honest. 

And I am going to be supporting your bill, Mr. Chairman, obvi- 
ously, and I want us to be able to get the open and reviewable 
science and the methodology that was used. We are dealing on an- 
other issue on flood on FEMA where we are questioning their data. 
But I think that it would be a good thing for all scientists to be 
able to review the data or am I missing something here? Mr. 
Keating, I will start with you. 

Mr. Keating. I think you are spot on. And what is interesting 
about this, this is where public choice economists can help out a 
little bit here because they are the ones that point out what the 
incentives are in government, right? What are the political incen- 
tives? And free-market people, you know, we still have a certain 
skepticism, which I like, regarding what government does. So what 
you want to do when you are talking about science is clear that up 
as much as you can, dispel the idea that this is all politics, it is 
all driven by politics 

Mr. Weber. And I apologize. I am running out of time. If I may 
go over 

Mr. Keating. Yes. 

Mr. Weber. — to the good doctor next to you here. Same ques- 
tion. I think it is a good thing if most all of the studies were up 
for review by all scientists involved, don’t you agree? 

Ms. SiLBERGELD. I think we have that process in place now, sir. 
And actually, if I may, in response — I am sorry Dr. Broun is not 
here because in fact his profession has driven what I think to be 
a model, which is the evidence-based approach. And physicians 
don’t actually usually consult the underlying data. They look at 
systematic reviews 

Mr. Weber. And, I am sorry. I need to move on. Dr. Cox? 

Dr. Cox. I think it is an excellent idea and I don’t think that our 
current systems go far enough in the public interest. 

Mr. Weber. And, Dr. Graham, I think 

Dr. Graham. I agree. 

Mr. Weber. Okay. Well, we want to be very — I know — I believe 
that all of our colleagues here want to be good stewards of the 
American tax dollars or taxpayers’ money, and so the fact that we 
want to call in to question some of these policies by the EPA be- 
cause they are killing jobs; it is just as pure and simple as you can 
make it. And if the scientific data is not absolutely concrete and 
100 percent certain, then I think rather than have a war on coal, 
rather than have a war on fossil fuels — and I was on the Environ- 
mental Reg Committee in the Texas Legislature, and Texas does a 
good job of cleaning up their air. Now, there are federal govern- 
ment employees would like to think that they have to come in and 
riot herd on Texas to use an old Texas colloquialism. But leave it 
to the States for the most part because they do want a clean back- 
yard. 

And, Mr. Chairman, I have gone on way too long and I will stop. 
Thank you very much. 

Chairman SCHWEIKERT. Thank you, Mr. Weber. 
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Ms. Bonamici, do you ever feel that we are just surrounded by 
Texans on this Committee? 

Ms. Bonamici. Yes, indeed, Mr. Chairman. 

Chairman ScHWEiKERT. Would you like to do a closing statement 
or share some thoughts? 

Ms. Bonamici. Thank you very much, Mr. Chairman. I appre- 
ciate the opportunity. And thank you for your courtesy in allowing 
me to do this. But thank you to all the witnesses for being here 
today. 

Mr. Chairman, I know we share a common goal of transparency 
and I also — but the issue of course is how do we accomplish that? 
And I also know that we all want the EPA to be able to use the 
best science available and I look forward to further discussions 
about that. 

I also want to reiterate the need to hear from the EPA about this 
proposed legislation to get their input on how this would affect 
them, their work, their workload, and just to get a sense from them 
on the record and again working hopefully with the Subcommittee 
on Science and Technology. So I look forward to — the Sub- 
committee on Technology and Innovation. I look forward to working 
with you on this important issue, and as I said, we share the com- 
mon goal of transparency. Let’s figure out if there is a way we can 
get there. Thank you. 

Chairman SCHWEIKERT. Thank you, Ms. Bonamici. 

And this was one of those I have a particular personal interest 
in, so forgive me if sometimes — and I am hoping I get input from 
everyone, particularly those agencies that would be affected. 

There is a running joke in my family, what are the two times in 
life you think you know everything? When you are 14 years old and 
the day after you get elected to Congress. It really is actually 
funny. And my concern is, you know, in part of this discussion, 
should data that is making regulatory policy, how egalitarian, how 
much should it be? And, you know, we all have this certain sort 
of folklore, experience in our lives. When I was a freshman in my 
statistics class, my professor at that time, she talked about how 
she had done all this modeling on what — ^you know, for a couple 
drug companies on what the different products for ulcers would be, 
but a couple years later she found out that the ulcers she had were 
actually caused by bacteria in the lining. So all this study over here 
on what was the best drug, it turns out they were looking at the 
wrong thing. They had it wrong. 

How often does that happen where the data sets, our current 
data belief, our current policy we believe today will be dramatically 
different a decade ago? It was only 10, 12 years ago if you and I 
sat in this room, we would have been hearing speakers. Members 
talking about Peak Oil, you know, the next incremental barrel of 
fossil fuels would be less. We screwed up somewhere the modeling 
on the understanding of technological curve, where we were at. We 
got it wrong but yet our tax policy, our environmental policy, our 
military policy was based on that data. And I am — so part of my 
embracing of the idea of lots of inputs is I am hoping somewhere 
there is the brilliance that helps us do what is best, and what is 
best for our country sometimes may have my ideological leaning 
and sometimes it might not, but at least it will be fact-based. 



107 


So with that, I want to thank the witnesses here and I am very 
sincere. Having read all your CVs, you are all very, very unique in- 
dividuals and very bright. 

The Members of the Committee will have — if they have addi- 
tional questions for you, and I am almost sure there will be some 
coming towards you, we will ask you to respond in writing. The 
record will remain open for two weeks for additional comments and 
written questions from Members. And the witnesses are excused. 
Thank you for giving us some of your valuable time. 

[Whereupon, at 11:38 a.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by The Honorable John Graham 

U.S. HOUSE OF REPRESENTATIVES 
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 
Subcommittee on Environment 

Hearing Questions for the Record 
The Honorable David Schweikert 


Ensuring Open Science at EP A 

The Honorable John Graham 


I . Why do you think “the vast majority of the environmental scientific community should 
not have difficulty satisfying the public access provision” in the Secret Science Reform 
Act? 

Answer: Once environmental scientists have published their work in the peer-reviewed 
scientific community, it is already common practice for them to share their data with 
other scientists who have an interest in their research. The Secret Science Reform Act 
would simply add EPA scientists to the list of colleagues who may request access to 
their data. Once EPA scientists obtain the data, they can post it on their web site 
(barring any legal restrictions). 


2. You cite recent recommendations from the Administrative Conference of the U.S.as 
being supportive of the types of data U^nsparency required in the Secret Science 
Reform Act. Can you explain what the Conference does and how these 
recommendations relate to EPA’s approach to open science? 

Answer: The Administrative Conference of the United States is a nonpartisan 
organization charted and funded hy Congress with the mission of improving the 
transparency and quality of federal rulemaking in the United States. The 
recommendations made by ACS on data transparency are applicable to all federal 
agencies, including the EPA. 

3. In 2012, the President’s Science Advisor testified that “Absolutely, the data on which 
regulatory decisions... are based should be made available to the Committee and 
should be made public.” Also in 2012, the Chair of EPA’s Science Advisory Board 
testified that EPA’s advisors recommend “that literature and data used by EPA be peer- 
reviewed and made available to the public.” Do you agree with these statements? 

Answer: Yes 
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4. In analyzing the economic benefits of the Clean Air Act, EPA often cites a study it 
conducted that finds that the Act achieves $2 trillion in benefits from 1990 to 2020, 
with a benefits exceeding costs by a margin of 30-to-l. More than 80 percent of those 
benefits, though, are derived from studies on fine particulate matter, for which the 
underlying data has not been made public. Do you find this analysis credible? 

Answer: They are not credible without an appropriate indication of uncertainty. They 
are particularly vulnerable to mistrust among stakeholders because the underlying data 
have not been made public and thus are not available for verification and sensitivity 
analysis. 

5. You mentioned in your testimony that the data access provisions for the journal Science 
are consistent with the principles in the legislation. Similarly, the journal Nature also 
tells its authors: “An inherent principle of publication is that others should be able to 
replicate and build upon the authors’ published claims. Therefore, a condition of 
publications in a Nature journal is that authors are required to make materials, data and 
associated protocols promptly available to readers without undue qualifications.” Do 
you believe this legislation is consistent with the policies of these journals? 

Answer: Yes. 

6. You emphasize in your testimony the importance of ensuring transparency and access 
for assessments and guidance beyond just regulations, stating that “some of the most 
significant actions taken by EPA are scientific determinations rather than regulations 
per se.” Can you discuss the risks of these types of determinations and ways in which 
openness could improve these EPA products? 

Answer: When a federal agency makes a determination that a product, technology or 
substance is hazardous, the determination itself - without any formal regulatory action 
- can create a stigma in the marketplace that causes a loss of sales, jobs and so 
forth. The stigma can also trigger lawsuits against companies under the common laws 
of the fifty states. If the scientific and technical data underpinning the determinations 
are not transparent and reproducible, it can be quite difficult for scientists in an 
impacted company — or any scientist — to determine whether the determination is valid. 

7. Do you agree that the data transparency policies in the legislation should apply to both 
data and models, regardless of who funded them? 

Answer: Yes. 

8. In 2009, you were part of a Bipartisan Policy Center panel that unanimously 
recommended that “studies used in the formulation of regulation should be subject to 
data access requirements... regardless of who funded the study.” 

a. Is the Secret Science Reform Act consistent with this recommendation? 

Answer: Yes. 
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b. Since the legislation applies to any scientific information, regardless of who 

funded or carried out the research, do you think the bill addresses Dr. Silbergeld’s 
concern that industry fails to disclose critical information for EPA regulations? 

Answer: Yes. 


9. In your testimony on fine particulate matter, you noted some of limitations of the EPA- 
funded re-analysis of critical data. Do you agree with the National Academy of 
Sciences’ stated that “the Health Effects Institute reanalysis model is... not a substitute 
for public access to data”? 

Answer: Yes. 

10. During the hearing. Dr. Silbergeld referenced the model of evidence based decision 
making. In your view, are the transparency principles consistent with evidence based 
decision making? 

Answer: Yes. 


1 1. During the hearing. Dr. Silbergeld stated in during the hearing that, “[peer reviewers] 
do a very exhaustive job of reviewing.” In your view, are all peer reviewers provided 
access to data to ensure reproducibility? 

Answer: No. 

12. In your experience, does EPA’s communication of regulatory benefits include all 
associated uncertainties in their scientific findings? 

Answer: No. 
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Questions for the Record 


Submitted by: Full Committee Ranking Member Eddie Bernice Johnson 
Subcommittee on Environment 

Hearing: Ensuring Open Science at EPA — February 1 1 , 20 1 4 


Questions for Dr. John D. Graham 

1) In your testimony you indicate that the “original health data” from the Harvard Six-Cities 
Study and the American Cancer Society study is not publicly available to researchers 
through EPA’s website. 


a. Are you aware the EPA is not the legal custodian of the “original health data?” 
(Yes/No) No. 

b. Are you also aware that the rightful owners of the data, the American Cancer Society 
and Harvard University, have procedures in place to make this data available to 
legitimate researchers? (Yes/No) Yes. 

c. You mentioned that HEI conducted a third party examination of the studies and that 
no major errors were found. It is my understanding that HEI not only validated the 
results of the original studies, but found a stronger association between air quality and 
mortality. Do you disagree with HEI, and the way they analyzed their data? (Yes/No) 
No. 


2) In October of 1991 you solicited grant monies from Philip Morris for the Harvard Center 
for Risk Analysis. It should be noted that you addressed your grant solicitation to Philip 
Morris’ Vice President for Government Affairs, which seems like an odd choice for an 
academic institution seeking grant support. In 1992 Philip Morris sent the Harvard 
Center for Risk Analysis a check for $25,000. However, just a short time later Philip 
Morris put a stop payment order on the check. In that same year, Kraft General Foods, 
which was a subsidiary of Philip Morris, sent the Harvard Center for Risk Analysis a 
check for $20,000. It should be noted that in that letter several Philip Morris contacts you 
dealt with during the previous solicitation were cc’d. (Please see attached documents) 

Answer: Ido not recall the precise events or motives as they occurred more than 1 5 
years ago. However, I believe they are reviewed in the hearing record of my Senate 
confirmation to the position of OIRA Administrator. See Nominations of Angela B. 
Styles, Stephen A. Perry and John D. Graham. Hearing Before the Committee on 
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Govemmentai Affairs of the United States Senate. First Session. 107* Congress. 
Washington, DC. May 17,2001. 


a. Did you or an employee at the Harvard Center for Risk Analysis request that Philip 
Morris cancel their grant to the Harvard Center for Risk Analysis? 

b. Did you or an employee at the Harvard Center for Risk Analysis request that the grant 
instead come from Philip Morris subsidiary Kraft General Foods? 

c. Assuming that the money from Philip Morris was refused, as the documentary record 
shows, was the motive for refusing that funding the perceived stigma of accepting 
tobacco money? If not, please explain. 


3) As Dean of the University of Indiana School of Public and Environmental Affairs 
(SPEA) please describe the kind of ethics training that exists in your program. If any 
materials describe this training (syllabus, course descriptions, etc.) please provide those 
to the Committee. Please indicate what has been done in this area since you arrived. 

Answer; In our undergraduate and graduate programs, my understanding is that 
professional ethics are weaved into a variety of courses but we do not rely on any single 
course in professional ethics to cover this material. 

4) What rules does the School of Public and Environmental Affairs apply to faculty 
members in disclosing funding sources for their work when they discuss or publish that 
research? Please identify changes to those rules that have occurred at SPEA since you 
arrived. 

Answer: There have been no changes in disclosure rules at SPEA since I arrived in 
2008. Under sponsored research agreements with outside parties, faculty members are 
expected to disclose the funding source(s) in their peer-reviewed products. 

5) The legislation you testified upon, H.R. 4012, proposes measures that would 
fundamentally change what EPA would have to do with scientific information for any 
“risk, exposure, or hazard assessment, criteria document, standard, limitation, regulation, 
regulatory impact analysis, or guidance.” Given that EPA has jurisdiction over 
pestieides, water, air, surface pollution, superfund clean ups, chemical risk assessments 
and exposure guidance — just to name some of the issues at the Agency — there are 
innumerable regulated industries that would have some stake in seeing this measure move 
forward. You had a rich history of soliciting funding from regulated industry to support 
your work when you were at the Harvard Center for Risk Assessment. As Dean of the 
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School of Public and Environmental Affairs, you probably have fund-raising 
responsibilities in your portfolio. 

a. Please identify all parties (to include third party organizations funded by regulated 
parties) that you have solicited support from that would have an interest in the 
changes proposed in H.R. 4012. 

b. Please identify all parties (including third party organizations funded by regulated 
parties) that have contributed to support the work of the School since 2008 that 
would have an interest in the changes proposed in H.R. 4012. 


Answer: Although I have seen claims in the media and at the hearing about organizations 
who may favor or oppose H.R. 4012, 1 am not aware of any specific organizations that 
would have an “interest” (meaning financial or commercial) in the specific changes 
proposed by H.R. 4012. Most of my fundraising activity as Dean is directed at individual 
philanthropists, and I am not aware of any of them who would have an interest in the 
changes proposed by H.R. 4012. 


6) H.R. 4012 reflects a tension between the desire to make science and data transparent, and 
the need to respect existing laws that protect individual interests — for example— patient 
and study participant privacy, confidential business information and intellectual property 
protections. On the one hand, EPA is prohibited in the text from using any “scientific 
and technical information” in a covered action unless that information is specifically 
identified, and publically available in a manner that is sufficient for independent analysis 
and substantial reproduction of results. On the other hand, the bill specifies that EPA is 
not required to engage in public dissemination of information “the disclosure of which is 
prohibited by law.” It is hard to see how EPA could rely upon most of the major public 
health research of the last 40 years if they must meet the first set of standards without 
violating the law by making privacy protected health information publicly available. 

a. You acknowledged in your oral testimony that the public health research that EPA 
relies upon is an area where patient sensitive information remains a barrier to full 
data sharing. However, you seemed to suggest that the issue is not difficult and 
resides more in the attitudes of the “mental health field” and the “environmental 
health research community” — by which I assume you meant the public health 
community — just need to be persuaded to get on board. Please explain in some 
detail how you believe data can be released in a way that allows for independent 
analysis and substantial reproducibility but does not violate individual privacy? 

Answer: There must be a misunderstanding or a miscommunication on my part. I 
did not intend to refer in any way to “mental health field”. Disclosures of data can 
protect privacy by removing identifiers (e.g., names, social security numbers, 
addresses). The Census Bureau regularly collects data from individuals and makes it 
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publicly available but does not violate the privacy of individuals. Dr. Cox also 
described in his testimony a variety of ways that this issue can be addressed, and the 
March 2014 Institute of Medicine Workshop is devoted precisely to this issue. 

b. Given that participants in health studies are always promised anonymity and 
privacy is assured by law, what would a “voluntary” breach of that privacy by a 
Government agency mean for future public health studies? 

Answer: I do not know. 

c. In your oral testimony you indicated that Confidential Business Information does 
not apply to “90, 95 percent of the industry information” that EPA relies upon 
based on your experience at OIRA. This estimate may have something to do with 
what OIRA actually reviewed, or what your staff shared with you. A vast range 
of EPA information has been subject to CBI: pesticide approvals all involve CBI 
information; costs of compliance often rely on company specific survey data; risk 
assessment information can include company, even factory-specific submissions 
about exposures; TSCA reports are full of CBI information and that data may be 
used in other proposals by the agency that would be a covered action under the 
bill. For the purposes of meeting the bill’s goal of providing all data necessary for 
independent analysis and substantial reproducibility, what proportion of the 
infomiation you were exposed to falls into that category and would be CBI? 
Please expand in detail on the specific EPA matters that you were able to review 
that lead you to this estimate. 

Answer; You are correct that OIRA reviews of EPA actions are concentrated more in 
the air and water offices than in the pesticide and toxics offices. However, as an 
academic I have written extensively on the processes used in pesticides and toxics 
offices and, even in those offices, much of the data submitted by industry is not 
confidential business information. For industry data supplied in air and water office 
rulemakings, very little is confidential business information. 


7) When you were at OIRA, you brought in a toxicologist, Nancy Beck. Early in the 

Obama Administration she left OIRA to go to work for the American Chemistry Counsel. 
Did you serve as a reference for Nancy Beck? 

Answer: I do not recall whether I served as a reference for Nancy Beck. 
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Responses by Cox Dr. Louis Anthony Cox, Jr. 

Cox Associates. 503 Franklin St, , Denver, CO, 8021 8. Ph 303-388-1 778; Fax 303-388-0609 
www.cox-associates.com 


Rep. David Schweikert March 13, 2014 

Chairman 

Environment Subcommittee 
Committee on Science, Space, and Technology 
2321 Rayburn House Office Building 
Washington, D.C. 20515-6301 

Dear Rep. Schweikert, 

I am pleased to provide the following responses to the questions submitted for the record by 
Members of the Committee, as set forth in the attachment to your letter dated February 27, 20 1 4. 

1 . Do you agree with the National Research Council, who in 2002, stated that “peer review does 
not detect fraud, validate factual findings. ..or substitute for the judgments of the scientific 
community as a whole?” 

Yes, I agree with the National Research Council’s statement. The high prevalence of 
false-positive results in published studies to which I have alluded, and to which I have 
provided references in my written testimony, applies to peer-reviewed studies. 

2. You have examined the relationship between fine particulate matter and mortality within a 
publicly available data set, coincidentally run by John Hopkins University. 

a. Does this data suggest a causal relationship between reducing fine particulate matter 
and reduced mortality? 

b. Because the data set you used is public, you suggest that “others will, repeat and 
improve upon ray analysis.” Is that true for the data that ERA relies upon in 
concluding that there is a causal relationship between fine particulate matter and 
mortality? 

(a) No, the NMMAPS data indicate a clear positive statistical association between 
PM2.5 levels and levels of mortality risk (e.g., both tend to be high on cold winter days), 
but they do not suggest any causal relation between reductions in PM2.5 and reductions 
in mortality risks. After controlling for temperature, month, and city, there is no 
evidence of any causal relation between PM2.5 levels and mortality risks. This finding 
is discussed further in the following references: , 

a. Cox LA Jr, Popken DA, Ricei PF. Warmer is healthier: effects on mortality rates 
of changes in average fine particulate matter fPM2.5) concentrations and 
temperatures in 100 U.S. cities. Regul Toxicol Pharmacol. 2013 Aug;66(3):336- 
46. doi: 10.1016/j.yrtph.2013.0S.006. Epub 2013 May 21. 

b. Cox T, Popken D, Ricci PF. Temperature. Not Fine Particulate Matter tPMZ.S), 
is Causally Associated with Short-Term Acute Daily Mortality Rates: Results 
from One Hundred United States Cities. Dose Response. 2012 Dec 14;11(3):319- 
43. doi: 10.2203/dose-response.l2-034.Cox. eCollection 2012. 
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3. In 20 1 2, the President’s Science Advisor testified that “Absolutely, the data on whieh 
regulatory decisions... are based should be made available to the Committee and should be 
made public.” Also in 2012, the Chair of EPA’s Science Advisory Board testified that 
EPA’s advisors recommend “that literature and data used by EPA be peer-reviewed and 
made available to the public.” Do you agree with these statements? 

Yes, I agree with these statements. 

4. In analyzing the economic benefits of the Clean Air Act, EPA often cites a study it conducted 
that finds that the Act achieves $2 trillion in benefits from 1990 to 2020, with a benefits 
exceeding costs by a margin of 30-to-l . More than 80 percent of those benefits, though, are 
derived from studies on fine particulate matter, for which the underlying data has not been 
made public. Do you find this analysis credible? 

No, I do not find EPA’s analysis or resulting estimate of benefits credible. It is based on 
unproved and scientifically implausible assumptions about concentration-response 
associations (entirely linear, entirely causal, no threshold, no heterogeneity), incorrect 
arithmetic for accounting for numbers of “avoided deaths” per year, questionable 
economic value judgments about the dollar value of “avoided deaths” that do not 
consider how long deaths are postponed; and an approach to uncertainty analysis that 
presupposes the answer (use of Weibull uncertainty distributions, which implicitly 
assume a 100% probability of a positive relation). These points are discussed further in 
the following reference: 

a. Cox LA Jr. Reassessing the human health benefits from cleaner air. Risk 
Analysis. 2012 May;32(S):816-29. doi: 10.1111/j.l539-6924.2011.01698.x. 

5. You discussed the ability to protect any confidential or individually identifiable information 
in data sets used by federal regulators. Are these methods cost-effective, and are there 
examples of government or academic data sets that follow these procedures now? 

Yes, cost-effective statistical methods for protecting individual privacy and 
confidentiality are well developed and have been applied to both government and 
academic data sets. My written testimony cited methods based on multiple imputation, 
used at the Census Bureau. A textbook on Statistical Disclosure Control, by Hundepool 
et al., with case studies thttD://www.wilev.comAVilevCDAAVilevTitle/DroductCd- 
1119978157.htm If provides further discussion of some of the cost-effective methods that 
are used to protect confidential data. Financial, medical, and other industries have 
developed working solutions for protecting confidential or individually identifiable 
information in data sets used by Federal regulators and others, and such methods have 
been extended specifically to environmental and health data over the past twenty years 
(e.g., Cox, LH, Protecting confidentiality in small nonulation health and environmental 
statistics. Stat Med. 1996 Sep 15-30;15(17-18):1895-905.). 
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6. You cite the unpredicted results of coal bans in Dublin as evidence of why data access is 
important. How do these findings relate to US EPA regulations? 

The coal-burning bans in Dublin were based on unjustified interpretation of positive 
statistical associations between exposure levels and mortality risk levels as evidence that 
reductions in exposures would cause reductions in mortality risks. Dr. Douglas 
Dockery of the Harvard School of Public Health, a prominent air pollution health 
effects researcher whose work and opinions about causation have informed US EPA’s 
beliefs and assumptions, championed this misleading causal interpretation of the 
statistical associations. This interpretation turned out to be unjustified. Yet, precisely 
such misinterpretation of exposure-response associations as if they were causal relations 
provides the foundation for the US EPA’s assessment of the projected public health 
benefits from tighter regulation of fine particulate matter and other pollutants. This 
reasoning was not sound in Dublin, and it is not any sounder in the US. What is 
particularly frustrating is that is obvious to many methodologists (some of whom I cited 
in my written testimony) that the types of causal inferences drawn by Dockery and 
colleagues and the US EPA are not valid, yet it is not easy to correct their calculations 
using appropriate methods of valid causal analysis and inference because it is not easy 
to gain access to their data. 

7. Can you identify recent developments in scientific journals trending more greater data access 
policies? 

Recent developments in scientific journals toward greater data access include the use of 
supplemental online information to provide access to original data and analyses; open 
data access policies; and improvements in public data archiving for scientific research 
(e.g., 

httD:/Avww.plosbiologv.org/article/info%3Adoi%2F10.1371%2Fiournal.Dbio.l001779> . 
For example, as of this month, “[A]ll PLOS journals will require that ail manuscripts 
have an accompanying data availability statement for the data used in that piece of 
research” thttp://bloEs.plos.org/bioloEue/2014/02/03/opening-up-data-access-not-iust- 
articles/ l. 

8. Dr. Silbergeld references the model of evidence based decision making. In your view, are the 
transparency principles in the Secret Science Reform Act consistent with evidence based 
decision making? 

Yes, in my view the transparency principles called for in the Secret Science Reform Act 
are consistent with, and would facilitate, evidence-based decision making. 


9. Dr. Silbergeld stated that, “[peer reviewers] do a very exhaustive job of reviewing.” In your 
view, are all peer reviewers provided access to data to ensure reproducibility? 
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No, I know from direct experience that peer reviewers are not usually provided access 
to data and cannot ensure reproducibility. Indeed, prestigious journals such as Science 
and Nature are in the process of changing their policies and editorial boards to increase 
reproducibility, due to the crisis of false positives and irreproducible results now being 
published in top-rated peer reviewed scientific journals. 

10. Dr. Silbergeld stated that the bill, “eliminate a great deal of the epidemiologic and clinical 
literature that could be available,” due to confidentiality concerns. Do you agree with this 
characterization? 

I disagree with Dr. Silbergeld that the bill would eliminate any of the epidemiological 
and clinical literature. What it would do is to allow the factual basis for claims in the 
literature to be scrutinized where practicable; for claims that cannot be independently 
verified to be identified as such; and for future publications and claims to be put on a 
more trustworthy basis in which both the public and other scientists are more easily 
able to determine whether and how well opinions and interpretations are actually 
backed by data, as opposed to being mere personal opinions or interpretations of the 
authors, masquerading as objective scientific findings. 

1 1. Dr. Silbergeld suggested that studies involving location of subjects could not be de-identifled 
in a way that allows data to be used by other scientists. Do you agree with this 
characterization? 

I disagree with Dr. Silbergeld’s characterization. Techniques such as geomasking are 
now well developed and inexpensive to apply. They can be used to make spatial data 
available for analysis while protecting individual confidentiality. The following 
references provide points of entry to this body of knowledge. 

a. Wieland SC, Cassa CA, Mandl KD, Berger B. Revealing the spatial distribution 
of a disease while preserving privacy. Proc Natl Acad Sci USA. 2008 Nov 
18;105(46):17608-13. doi: 10.1073/pnas.0801021105. 

b. Hampton KH, Fitch MK, Allshouse WB, Doherty lA, Gesink DC, Leone PA, 
Serre ML, Miller WC. Mapping health data: improved privacy protection with 
donut method geomasking. Am J Epidemiol. 2010 Nov l;172(9):1062-9. doi: 
10.1093/aje/kwq248 

12. In your experience, does EPA’s communication of regulatory benefits include all associated 
uncertainties in their scientific findings? 

In my experience, EPA’s communication of regulatory benefits conceals, rather than 
reveals, most of the key uncertainties in their scientific findings. EPA typically 
communicates regulatory benefits using point estimates (or best estimates) and 
statistical confidence bands or uncertainty intervals around the estimates. This fails to 
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communicate major discrete uncertainties, such as about whether the underlying 
assumptions of the analysis are valid, the probability that there are no benefits at all, 
the probability that the statistical associations used to estimate benefits are not causal, 
and the probability that further reductions in air pollution levels will produce few or no 
benefits because currently permitted levels are already low enough so that they do not 
cause the adverse health effects attributed to them based on historical data. Further 
discussion of EPA’s approach to quantifying and communicating uncertainty in 
regulatory benefits estimates, and why it is misleading, is given in Cox LA Jr. 
Reassessing the human health benefits from cleaner air. Risk Analysis. 2012 
May;32(5):816-29. doi: 10.1111/j.I539-6924.201I.01698.x. 
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Responses by Dr. Ellen Silbergeld 
March 4,2014 

RESPONSE TO COMMITTEE LETTER: 

Dr Ellen K Silbergeld, Johns Hopkins 

Cong. Johnson 

1 . In your written testimony you mention the need to improve the disclosure of industry data 
and rightfully point out that the “Secret Science Reform Act of 2014” seems to take a 
one-sided approach by targeting data from the academic community. You state, “the 
proposed bill would continue to immunize industry from disclosure while increasing the 
burden on EPA and, by pass through, on non-industry researchers.” 

a. Can you please describe the challenges associated with gaining access to industry 
data? 

b. How does limited access to industry data effect your ability as a scientist to reach 
evidence based decisions relevant to human health? 

1. (a) challenges associated with gaining access to indnstry data: 

These challenges and adverse impacts of inaccessible industry data are well known and 
were in fact the driver for the High Prodnction Volnme Chemicals project developed by 
governments, industry, and NGOs in the OECD Environment Program, as discussed in my 
testimony. There are foor types of barriers to accessing industry data: (1) overly eipansive 
definitions of “confidential bnsiness information” that preclode accessing information 
relevant to health and safety [most egregiously with respect to pesticides and biocides test 
data]; (2) inadequate data disclosures by industry in many MSDS documents (e.g., WV 
spill] and limited disclosures by industry in the Tonic Release Inventory Program at EPA; 
(3) deliberate efforts by US industry to block other programs that generate information, 
such as the CDC bionionitoring program, the Annual Report on Carcinogens hy NIH, and 
US industry lobbying in opposition to the REACH program of chemical regulation in the 
EU; and (4) lapses in product stewardship (as defined by the American Chemistry Councii) 
such that industry does not adequately test chemical constituents of consumer products, 
stored materials, and releases. 

(b) Impact of limited access 

Without information it is not possible to utilize evidence based approaches to decision 
making in environmental health. Moreover, lack of information constrains inferences 
about potential harms and risks from exposures to structurally similar classes of 
compounds (such as flame retardants and endocrine disrupters). 

2. Dr. Graham seems to suggest in his testimony that EPA can simply rely on different 
sources of scientific and technical information for covered actions, if the agency cannot 
compel the release of data from the scientific community. This is perhaps the clearest 
language to indicate that the legislation would constrain EPA, limiting the scope of 
science available to the agency. 


1 



a. Can you please discuss the implications these restrictions would have on researchers 
and the quality of work conducted by the agency? 

b. How important is it that EPA have the ability to consider the best available science to 
inform its regulatory actions? 

(a) restrictions on EPA use of scientific information 

If the EPA is unable to access the peer reviewed literature because raw data are not 
available as proposed in the “Secret Science^’ bill, then we move to the dysfunction situation 
where the EPA will be unable to sustain its decisions because these will be based on 
inadequate or incomplete science. 

(b) Importance of best available science for EPA decision making 

It is indeed quite strange why those parties who in the past have criticized the EPA for not 
relying on “good science” now seek to prevent EPA from utiliziDg all relevant and “good” 
science by setting up obstacles to its generation and use. 
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Cong Bonamici: 

1 . In your testimony, you discuss the use of third party review as a means of enabling data 
re-analysis in a non-adversarial way. In fact, as recognized by the other witnesses at the 
hearing, this type of arrangement was used to re-analyze the data from the Harvard Six 
Cities Studies and the American Cancer Society Cohort data, the studies often referred to 
as “secret science.” 

a) Please discuss the use of third party review and explain why it is an effective 
model. 

b) Did the HEI (Health Effects Institute) re-analysis of the Harvard and American 
Cancer Society studies overturn the conclusions of the original articles? 

c) In cases where it is impossible to remove all of the identifying data without 
negating the scientific value of the dataset, is it still appropriate for EPA to use 
this research to inform its regulatory decision making? If so, why? 

1 . 

(a) third party review of data 

When there are supportable reasons for re-analysis of primary data, there are established 
methods of third party review for accomplishing this goal [note that this differs from 
demanding universal access to all such data in the absence of any demonstrated need for 
access or re-analysis]. Third party review by definition involves outside experts with no 
connection to the original researchers, to the EPA or to any stakeholder demanding such 
review. Third party review has proven to be elective because of its focus and coherent 
basis. Such analyses have been designed to evaluate specific issues or findings rather than 
unfocussed “data dredging’*^ Because of the limited scope of third party review and the 
engagement of reputable and non-adversarial reviewers, this method has protected 
participant confidentiality and other ethical issues. 

(b) HEI review of Harvard and ACS studies 

A copy of the review summary is attached. To quote the findings of the HEI report; 
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RESULTS AND IMPUCATJONS 


PART 1: REPUCATION AND VALIDATION 
« An fixiensive audit of tbe study peculation 
data fiv bath the Six Gties and A(2 Studies 
and of the air quality data in the Six Cities 
Study revealed the data to be of generally hi^ 
quality with a fewexceptioos^ Inbotb studies, 
a few emus vvo-e found in tbe endiag and 
inclusion of cotain sid^ects: when those sub- 
iects were included in the amlyses, they did 
ncM niateriaUy chat^ the results as ariginaliy 
reported. Because the air quality data used in 
the ACS Study could not te audited, a sepa- 
nric air quality database was constnicted fur 
the sensitiviTy analyses described to Rirt IL 
• TheReanatj'sblbaniwasablBtoreplfciteilie 
affinal resulte io both stuches usii^ the same 

datn at|fl rtaticrifal Twrihnri^ m iwtyi hy thw Hrip- 

inal Inrest^alres. The Rfianah^ Thatn am* 
&nied the artgmal poiiil estinutes: F<t (he Six 


Chiss Study, they repexted the relalix'e risk of 
mnitality fran all causes assodaied with an 
iiK3«asa io Bse particiBs of 1&.6 |igAn^ as 1.26. 
dose to tbs 1.26 reported by the oHginal Ihres- 
dj^lOR. For the ACS Stuify. the retative risk of 
mistalily frcxn all causes assodaied willi an 
iDoease iDfiiiepaitidB5cif24.5 wasl.ia 
in the reanah'sis. dise to the 1.17 looted by 
the Original Investi^nis. 


(c) If EPA cannot remove all identifying data without negating a dataset. Is it still 
appropriate for EPA to utilize a study? 

In my opinion (see above) this is not a relevant question. It is rarely the case that access to 
original study data is necessary for EPA to utilize relevant scientific studies and for the 
public to have confidence in these studies as bases for regulatory' decisions. When such 
access is needed — which has been relatively rare — then strategies such as third party review 
can be employed. 

2. In your testimony you indicate that even though you support increasing transparency for 
information used to develop federal regulations, you recognize that the release of raw 
data associated with human studies could have potential ‘‘chilling” effects on “the 
engagement of the global scientific community in research relevant to the protection of 
human health and the environment.” 

a) Please describe the potential repercussions of EPA providing raw data from every 
study used to support a regulatory assessment. 

b) In your testimony you mention evidence-based toxicology. What is evidence- 
based toxicology? How can this improve the science basis for decision making at 
EPA? 

2. (a) repercussions of EPA being forced to provide raw data from every study 

This requirement will definitely add to further delay in reaching regulatory decisions by 
EPA. It will also have a chilling effect on the scientific community as a source of research, 
as Prof Lynn Goldman and I wrote in our commentary on this topic (paper attached to 
testimony). Since scientists such as me do not undertake research for the express purpose 
of or in anticipation of its use in regulatory proceedings, we cannot anticipate that such 
demands might be imposed upon us as individual researchers or on our institutions. 
Moreover, prior to such demands, we will have conducted epidemiological studies with 
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approval of our institutional review boards for human subjects research, and thus may find 
ourselves in a highly compromised and potentially unethical position of having to abrogate 
our declaration to participants in our studies that they will be protected from identiflcation 
or other disclosure. 

(b) What is evidence based toxicology? 

I appended a copy of my recent paper on this topic to the testimony. In that paper, Dr 
Roberta Scherer (of the Cochrane Collaboration US Coordinating Center) and I defined 
evidence based methods as: 

Evidence can be defined as the translation of information into accepted practice using 
methods that reduce bias and increase confidence. As in the law, evidence based methods 
involve the evaluation of information for its admission into consideration in decision 
making through the process of applying specified norms and methods. In order to avoid 
bias, these norms and methods stand apart from the information under consideration and 
their application is undertaken with complete transparency. 

Evidence based methods were first developed in medicine and then expanded to the broad 
domain of health care, including clinical trials of intervention efficacy, observational 
epidemiological studies on risk factors for disease, and diagnostic methods. We propose 
adoption of evidence based methods in toxicology, specifically the norms of transparency 
and the methodology of systematic reviews, which clearly incorporate the following steps: 
development and explicit framing of research questions that can be answered by a 
systematic review, and explicit statement of a publically available protocol for conducting 
the s^'stematk review. This protocol includes a defined and annotated strategy for locating 
sources of evidence; a priori conditions for exclusion and inclusion; defined analytic 
procedures to evaluate study designs and statistical methods; criteria for evaluating selected 
studies; methods for integrating study results. These rules are based on the assumption 
that all studies are well intentioned but no study is perfect. The goal is to identify all 
relevant sources of information in an unbiased manner and then to screen this body of 
information by identifying aspects of each study that can increase bias or uncertainty and to 
consider the impact of these aspects on analytic confidence. 

Experience over the past decades in application of evidence-based approaches in medicine 
and health care demonstrates that these methods can enhance decision making by 
increasing transparency of process and methods, by reducing bias in selecting and 
evaluating evidence, and by contributing to the improvement of methods used in toxicology 
and other disciplines to develop information relevant to environmental health and 
ecosystem decision making. 
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3, Do you agree with the assertion made by a number of witnesses at the hearing that there 
is minimal evidence for the economic benefit of EPA regulatory action, particularly 
actions limiting the emission of air toxins? If not, why? 


Evidence for the economic benefits of EPA regulatory particularly with respect to air toxics 


I do uot agree with this statemeut on the basis of the evidence. The costs of disease and 
dysfunctions associated with exposures to toxic chemicals in the environment are very high. 
The economic benefits for reducing these exposures are also very large and have been 
empirically demonstrated by a rich area of research and documentation. Overall, an 
important analysis of the beneficial impacts of air toxics regulation and enforcement was 
published in the NEJM by Choksi and Farley (2012) [copy attached]. As shown below, the 
overall cost effectiveness of environmental interventions (through regulation) exceeded the 
combined benefits of all other prevention modalities used in medicine. 



I Environmental 3 Nonclinical. person- B Oinicai intervention 
intervention directed Irrtervcntion 


C^'Effecthrencss of Categories of Piwe n thre Interventions. 


The authors concluded that “The scientific literature now points to the value of 
implementing preventive environmental interventions that are cost-saving”. 

Dr Richard Morgenstern of Resources for the Future (and formerly a member of the EPA’s 
Office of Policy Analysis) has also reviewed this topic from the perspective of an economist 
(for example, Krupnick and Morgenstern 2003, attached). 

There is also a large literature on the health and economic benefits of reducing 
concentrations of lead in ambient air, as well as reductions in ozone, SOx, and suspended 
particulate matter. For example, CDC published an analysis of temporal associations 
between lead in air (related to use of TEL in gasoline] and children’s exposure (blood lead) 
over the period of EPA mandated reductions in TEL use, as shown below. 
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LMd liMd hi GstoIlM (100 moirte loot/doy) 

These reductions were associated with reduced impacts on children’s nenrobehavioral 
development and school performance (Silbergcld 1997, attached). Additional benefits from 
these regulations continued to accrue through the phase-out of TEL by EPA regulation. 
Adults in the US have also benefitted from regulated reductions of lead in terms of reduced 
risks of cardiovascular disease and stroke as air lead levels have decreased. 

The benefits of regulated reductions in air pollutants have been analyzed by Correia et al 
2013, [paper attached]. There is a significant correiation between these reductions and 
increased life expectancy over the period from 2002 to 2007. The authors also found that 
reductions in air pollutants contributed also 20% to the increased life expectancy measured 
over this same period. 

Another approach might be to compare the associations between air pollution and health in 
China and the US, to utilize a “natural experiment” in lack of environmental regulation. 
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Abstract 

Lead poisoning is the most significant and prevalent disease of environmental 
origin among US childreni Despite over 100 years’ knowledge of the special 
hazards of lead exposure for young children, it has taken over a century for ef- 
fective primary prevention to be adopted. Obstacles to primary prevention have 
included deliberate campaigns by industry to prevent restrictions upon such uses 
on lead as plumbing, paints, and gasoline additives; influence of industrial support 
of biomedical research at major US medical schools; lack of appropriate policy 
mechanisms to identify and control lead exposures; and opposition to investing 
resources in lead poisoning prevention. The removal of lead from gasoline, which 
began in the United States in 1972 and was completed in 1995, has resulted in 
almost fourfold reductions in median blood lead levels in US children from 1976 
to 1 99 1 . Increased screening and interventions to identify and abate lead sources, 
such as lead in housing, also contributed to this major public health success. Nev- 
ertheless, lead exposures remain prevalent, although increasingly less generally 
distributed Perhaps because of the renewed “ghettoization” of lead, support for 
lead poisoning prevention has waned. Objections to investing public and pri- 
vate resources in screening and source abatement have challenged the continuing 
commitment of public health officials to prevention. The demonstrable success 
and social benefits of preventing lead toxicity are cited in support of continued 
preventive health policies. 


INTRODUCTION 

Lead poisoning, for much of this century, has been the most prevalent and 
serious disease of environmental origin for young children in the United States. 
Its causes were described accurately by the mid- 1920s, but not until the late 
1970s were comprehensive and effective actions taken to prevent this disease 
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by interdicting major uses of lead in consumer products and industry. The 
fact that over half a century elapsed between knowledge and action indicates 
that any review of the struggle to prevent lead poisoning in children must 
attend to the political and economic forces that for 50 years interfered with the 
implementation of primary prevention policies. The struggle for prevention 
has involved almost every consumer and environmental protection statute and 
policy strategy extant in the United States; it has engaged communities, states, 
and the federal government; it has revealed the nature and scope of injustice 
in the distribution of environmental risks; it has elicited courageous acts and 
great mendacity. From the perspective of a century’s experience, this review 
not only describes this history but also provides some insights that may be 
useful in preventing other environmental diseases, avoiding both the social toll 
exacted by lead and the political delays involved. Since the author of this review 
has been personally involved in much of the policy debate on lead poisoning 
prevention at the local, state, and federal over the past 20 years, this review 
should be read in light of the strong opinions held and expressed (65). 

No small part of the delay in recognizing the nature and scope of lead poi- 
soning and seizing opportunities for disease prex'eniion is due to the actions 
and expressed opinions of some biomedical researchers considered experts in 
their day. These scientists and clinicians — among them, Joseph Aub and Cecil 
Drinker of Harvard and Robert Kehoe of Cincinnati — were retained by the lead 
industry in its organized campaign to control the discourse on lead poisoning 
(see 29, 25, 55). For decades, from the late 1920s through the 1970s, the lead 
industry funded most of the research on lead toxicity at major institutions in 
the United States, including Harvard, the University of Cincinnati, and Johns 
Hopkins. Only with the rise of the National Institute of Environmental Health 
Sciences, under the leadership of David Rail, were other funds available, from 
a neutral source, and it is not coincidental that with this support a new genera- 
tion of researchers began to publish new findings on the toxicity and sources of 
lead in the mid-1970s. Even then, the struggle was neither easy nor pleasant. 
Attacks on the integrity of researchers reached their nadir when allegations 
of scientific fraud were raised by consultants to the lead industry against an 
internationally recognized scientist (64). Although c\entualiy rebutted, these 
charges were of a piece with earlier personal attacks by the industry and its 
consultants on Alice Hamilton and Randolph Byers. 

The history of lead must induce careful reflection: The natural and laudable 
bias of scientists toward vigorous skepticism can be exploited to support in- 
action on the basis of uncertainty. In public health, giving the “benefit of the 
doubt” to a potentially toxic substance like lead runs the risk of denying a mar- 
gin of protection to its victims. In the case of lead, the expressed uncertainties 
about low-level lead exposures (a relative term that changed over the course of 
this history, as described below) provided not only repeated excuses for inaction 
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LEAD POISONING PREVENTION 1 89 

but, more dangerously, justifications for new and expanded uses of this toxic 
metal. By the time these uncertainties were reduced to a level to support a 
new consensus on lead’s hazards in the early 1980s, thousands of tons of lead 
had been dispersed into the environment The responsibility for this catas- 
trophic mistake (to use the terms of Carl Shy) must be shared among industry, 
government, and academic researchers. 

The Nature of Childhood Lead Poisoning 

Lead poisoning is an entirely preventable disease, induced by exposures to 
lead. The toxicity of lead has been recognized for almost as long as this useful 
metal has been mined, smelted, and used by human societies (34, 42, 48). 
Rules prohibiting the use of lead additives in beverages were among the earliest 
food regulations in Europe and the American colonies (18, 75). Although it 
was recognized as early as the mid-nineteenth century that children might be 
among the most vulnerable to lead toxicity [for discussion of early opinions and 
observations, see Reference (50)], specific attention to children as a population 
at risk was only formalized in the 1920s. Hamilton (26) directed attention 
to the devastations of occupational and peri-occupational exposures for young 
children; she recommended that young children be excluded from employment 
in the lead trades, and that measures be taken to reduce the transport of lead 
dusts from the workplace to the home. In 1897, the Australian pediatrician 
Lockhart Gibson reported on cases of young children intoxicated by lead from 
lead-based paints used on porches and doors (24). The importance of lead paint 
as a cause of lead toxicity in children was soon reported in the United States, in 
Baltimore and in Boston [see (54) for a history of the recognition of lead paint 
poisoning in the United States]. 

The definition of lead poisoning — that is, at what level of exposure clinical 
toxicity occurs — has changed markedly over the century. Over this period, 
medical and public health opinion has shifted dramatically, from assuming 
that only high-dose, overtly encephalopaihic exposures were significant, to 
the recognition that very low doses, without clear symptom presentation, are 
associated with measurable neurotoxicity. 

Similarly, concepts related to treating lead poisoning have also undergone 
profound change. Prior to 1940, it was generally assumed that if the child’s 
exposure was promptly reduced, then the effects of lead were reversible unless 
severe toxicity had been induced (such as coma and convulsions). After the 
introduction of the chelating agents ethylenediamine tetraacetic acid (EDTA), 
d-penicillamine, and British anti-Lewisite (BAL) into therapeutic practice in 
the 1 950s and 1 960s, drug treatment was added to reduction of exposure as the 
means of medical management (13). Although there is a large medical liter- 
ature associating chelation treatment with reductions in blood lead levels and 
increased excretion of lead in urine, there is much less evidence that treatment 
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actually affects outcome. Silbei^eld & Chisolm (67) reported that children 
given EDTA showed reductions in blood lead and urinary catecholamine excre- 
tion, a potential marker of neurotoxicity; Rosen et al (56) reported that chelation 
lowered bone lead and improved children’s performance on measurements of 
cognitive function; and David et al (17) reported that chelation reduced hyper- 
activity in children with past lead exposure. However, these studies involved 
children with relatively high exposures to lead (in excess of 45 /rg/dL). 

The reversibility of lead toxicity has been questioned since Byers & Lord 
(9) suggested in 1943 that lead could induce persistent damage in children, 
even after their external exposures were reduced and clinical indicators of tox- 
icity were no longer detectable. Needleman and colleagues (45) demonstrated 
through a careful prospective study that early lead exposure affected children’s 
behavior and intellectual attainment for at least 10 years. Thus it is not clear 
how much benefit, in terms of reversing toxicity, any postexposure treatments 
bring. Chelation is undoubtedly valuable to lower high blood levels of and pre- 
vent acute neurotoxic effects, such as convulsions, as well as reducing effects 
of lead on kidney from prolonged high-dose exposure. Extended chelation can 
reduce overall body burdens of lead as well, including lead in bone (51, 57), 
Nutrition may play a role in reducing absorption of lead (27), but there is little 
evidence that essential elements, such as calciuin or iron, can counteract the 
presence of lead at target sites of action once it has been absorbed. 

These findings have had profound impacts upon strategies for prevention. 
It is now the general foundation for public health policy in the United States, 
France, Australia, and other countries that childhood lead poisoning, or toxicity, 
is associated with blood lead levels as low as 10 mcg/dL (or 0.5 mmol/L), and 
that the prevention of toxicity requires prevention of exposure through the iden- 
tification and control of lead sources in the environment, in air, food, water, and 
dusts and soils (see 50, 65 for reviews of national policies). The considerable 
decreases in what have been considered “acceptable” levels of lead exposure 
for children have driven an expanded consideration of opportunities to prevent 
exposures through more comprehensive identification and control of potential 
sources. Thus, the struggle to prevent lead poisoning has been transformed 
from tertiary/secondary prevention to primary prevention. Nevertheless, cur- 
rent public health policies remain imperfect instruments for achieving progress 
in primary prevention. In large part, this relates to the complexity of source 
reduction and to continuing controversies over the value of such strategies as 
universal screening. 

The Demographics of Lead Poisoning 

-The prevalence of childhood lead poisoning is largely determined by two fee- 
tors: age and proximity to environmental sources or media contaminated by 
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lead. Other factors that have been associated with increased levels of lead in 
blood — sex, income, race, place of residence — are mostly predictors of expo- 
sure. Children’s blood (PbB) lead levels are generally highest between 9 and 
18 months of age (see 2, 8). This is because children in this age range tend to 
explore their immediate environments intensively, with a good deal of hand-to- 
mouth activity (1 1, 12), and as compared to adults they are also more efficient, 
by five- to tenfold, at absorbing lead taken in orally (71). Some children have 
an abnormal pattern of ingesting non-food items, often soils (pica), which, if 
these soils are contaminated by lead, can result in very high doses of lead. Boys 
have slightly higher levels of PbB than girls, which is thought to be behavioral, 
related to a greater fi-equency of exploratory behaviors in early childhood. Nu- 
trition and genotype play a very slight role in modifying PbB (27, 77). 

The variables of income, race, and residence signify the social tragedy of lead 
poisoning in the United States, The poor and disadvantaged are more likely to 
live in lead-contaminated environments, especially in dilapidated housing with 
flagrant lead paint hazards; they are more likely to live in urban neighborhoods 
where years of traffic have left tons of lead deposits fi'om leaded gasoline; they 
are more likely to live near point sources of lead, such as smelters, or hazardous 
waste sites (41). 

The fact that lead poisoning is not evenly or randomly distributed among 
children in the United States continues to raise tremendous obstacles to its pre- 
vention. Until the late 1970s, it is fair to say that this disease was “ghettoized”, 
considered to be a risk exclusively to the urban minority poor (35, 43). This 
assumption was based on very limited data and highly skewed attempts to iden- 
tify children with elevated blood lead (PbB) levels. Even in Baltimore, the city 
with the longest history of public health surveillance for lead exposures, few 
children were tested or evaluated for lead exposure, and hardly any testing was 
conducted outside the well-defined “lead belt” of inncr-city housing, first de- 
scribed by Huntington Williams in the 1 930s (78). Concerns for lead poisoning 
waxed and waned with social concerns for the disadvantaged until the results of 
the first representative national surv ey of lead exposures in US children began 
to be published in 1982 (36). Coupled with the newer experimental and clinical 
studies demonstrating toxicity at lower levels of lead exposure (42), these na- 
tional prevalence reports substantially transformed lead poisoning as a public 
health issue by making it an environmental concern for th e general public (for in- 
stance, Enviromnental Defense Fund). Although the median PbB of poor urban 
black children measured in NHANES II (1976-1980) was the highest of all per- 
sons (nonoccupaiionally exposed), a significant proportion of oil children had 
PbB > 15 ^g/dLandthe median PbB for affluent white children was 13 ^g/dL. 

These data prompted efforts to prevent lead poisoning in children on a na- 
tional scale. 
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LEAD POISONING PREVENTION FROM 1900 TO 1970 

In the absence of national legislation or suitable government institutions, at- 
tempts to prevent lead poisoning over the first 70 years of this century (and ear- 
lier) were undertaken only by states and cities. Certain uses of lead, such as lead 
adulterants to cider and wine, were recognized to pose risks of excess exposure, 
and these were restricted on a local basis in Europe and the American colonies 
(18, 75). Other proposals — such as banning lead in plumbing, proposed as early 
as the eighteenth century in England (32) — ^were resisted by the lead industry, 
which in 1922 organized itself to fight attempts to restrict its products (55). The 
Lead Industries Association succeeded for years in stopping many regulations 
on lead, including blocking the United States from signing the International 
Labor Organization (ILO) convention prohibiting use of white lead in paint, an 
international treaty that the United States has never signed (48) (Table I). 

Most attention until 1970 focused, with reason, on occupational lead poison- 
ing. As documented by Hamilton (26), thousands of workers aimually were 


Table 1 Signatures to White Lead Paint Convention, International 
Labor Organization® 


1 923 Czechoslavakia, Sweden 

1924 Austria, Poland, Spain 

1925 Bulgaria, Chile, Romania 

1926 Belgium, France, Greece 

1928 Cuba, Luxembourg 

1 929 Finland, Norway, Yugoslavia 

1933 Columbia, Nicaragua, Uruguay, Venezuela 
1936 Argentina 

1938 Mexico 

1939 Afghanistan, The Netherlands 

1952 Italy 

1953 Vietnam 

1956 Hungary, Morocco, Tunisia 

1959 Guinea 

1960 Benin, Burkina Faso, Cameroon, Central African Republic, 

Chad, Congo, Gabon, Ivory Coast 

1 96 1 Mauritania, Niger 

1962 Algeria 

1964 Lao People’s Democratic Republic 

1966 Iraq 

1969 Democratic Kampuchea 

1970 Panama 

1976 Suriname 

1978 Comoros, Djibouti 

1988 Malta 


“Source: United Nations Environment Programme, 1989 (country names used 
are those shown in the source cited). 
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LEAD POISONING PREVENTION 1 93 

poisoned by lead in industries ranging from ceramics through battery manufac- 
ture. She also reported on the poisoning of children through their employment 
in the lead trades, and through the uncontrolled movement of lead dusts from 
the workplace to the worker's home, which remains a continuing problem (37). 
Some children were screened, starting in Baltimore in 1 935, with the develop- 
ment of laboratory methods at Johns Hopkins (31). But from 1935 to 1951, 
fewer than 2000 children were tested with the dithizone method (78). 

No actions were taken during this time on lead in environmental media such 
as air or water, in the absence of any comprehensive environmental legislation. 
An attempt was made in the 1930s to reduce allowable residues of lead in food 
(lead arsenate was widely used as a pesticide), but this was defeated by heavy 
industry lobbying. Tbe production and consumption of lead in the United States 
continued to grow. Worse, in 1925, a new use of lead was approved, and this 
was to become the largest single source of lead contamination in history. 

The decision to permit the use of alkyl lead additives in automobile friels 
represents the greatest single failure in preventing lead poisoning. Described 
as “the public health catastrophe of the 20th century” (63), this decision caused 
the release of thousands of metric tons of lead into ambient air, with both 
long- and short-range impacts. As shown in Figure 1, the greatest increase in 
human use of lead occurred at this point in time (note log scale on y-axis). 
This increase was quickly followed by rises in lead concentrations measured 
in stable ecological indicators, such as Greenland ice (42, 30). How the lead, 
gasoline, and automobile industries convinced the US Public Health Service to 
approve lead for gasoline has been eloquently described (58, 47). A sample of 
the industry propaganda of the time is shown in Figure 2. 

Despite the objections of Hamilton and others, a US government commission 
decided in 1925 to permit tetraethyl lead (TEL) use in gasoline, with a promise 
to reconsider the issue later (58). Unfortunately for the nation’s health, “later” 
took 50 years to occur. 

LEAD POISONING PREVENTION SINCE 1970 

This phase of the prevention struggle has been source directed, rather than case 
oriented. For the first time, dedicated efforts were made to identify and prevent 
lead exposures prior to exposure of children. These efforts took two forms: 
setting enforceable standards for environmental media and drinking water, and 
specific restrictions on certain uses of lead. Such actions were possible because 
of the creation of new government institutions, the Environmental Protection 
Agency (EPA) and the Occupational Safety and Health Administration (OSHA). 
These actions are summarized in Table 2 (50). Sometimes these actions flowed 
together. For instance, the EPA’s review of the National Ambient Air Quality 
Standard (NAAQS) for lead, begun in 1981 and concluded in 1986 (71), did 
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Figure / (/4) Hislorical trends in the use of lead; (5) concentrations of lead deposited in Greenland 
ice; and (C) the relative increase in lead over natural background levels. B.P., before present. Data 
fromNRC 1993. 


not result in any change in the NAAQS [although the EPA Science Advisory 
Board did recommend a reduction from 1.5 fig/rn^ to 0.5 or 0.75 (72)]; 

rather, its four volumes provided substantial weight to public campaigns to ban 
the use of lead in gasoline, a goal finally achieved in December 1995, after the 
passage of the Clean Air Act Amendments of 1990. In a similar fashion, the 
EPA’s decision to reduce the drinking water standard for lead from 50 ppb to 
5 ppb, in 1986, resulted in bans on use of lead in plumbing solders and fixtures 
to avoid exceedances due to leaching of lead into drinking water. 

These actions to improve environmental standards and to restrict lead uses 
had major effects on the nature and extent of lead poisoning in US children 
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Ethylized Gasoline 

F or one year and nine months ethylized gasoline has 
been on sale. It is now being distributed through 
about 20,000 filling stations covering one-third of the 
territory of the United States. 

About 200,000,000 gallons have been used by more than 
one million motorists, with complete safety and satisfaction. 

Recently a distre^ing accident occuned at an experimental 
plant, where a new process for the manu&cture of tetra-ethyl 
lead, one of the constituents used in ethylbJng gasoline, was 
under development. 

Tetra-ethyl lead is a poison, as ^ many raw materials 
which enter into the manufacture of harmless compounds. 

Ethylized gasoline consists of 1,300 parts of ordinary gasoline 
containing less than one pan of tetra-ethyl lead. 

This statement is issued to make plain the all-important 
difference between tetra-ethyl lead, the raw material, and 
ethylized ^soiinc, the commercial product 

Ethylized gasoline is more than an improved fuel. gKir^ 
smoothness to die motor and eliminating knocks; it i$ a 
scientific discovery which, iit its ultimate dex'elopment. will 
contribute largely to the conservation of the motU's supply 
of gasoline. 

The dangerous character of tetra-ethyl lead basing been 
recognized from (be outset, exhaustive tests have been con- 
ducted which have established the safety of ethylized ga»o- 
line when used properly as a motor fuel. 

These tests have been confirmed by the Linited States 
Bureau of Mines, which is making additional studies to de- 
termine whether any possible injury can result ftocn con- 
tinued contact when used for other than motor purposev. 

Scientific data based on these studies will be submitted to any 
health commissioner or other public health official on request. 

Figure 2 Advertisement placed by the Ethyl Gas Corporation, in the Baltimore Sun, in 1925, in 
the midst of controversy related to the risks of lead additives to gasoline. 


over the 1980s. The reduction in lead used in gasoline (greater than 100-fold, 
from 1976 to 1986) was associated with average reductions in PbB of over 
40%, as shown in Table 3 (66). Controls on lead in other products, such as 
increased vigilance by the FDA and CPSC over imported foods and toys, also 
probably contributed to these substantial reductions (2, 53). Use of lead in 
house paint had been restricted by regulation in 1977, but, for reasons dis- 
cussed below, this action has had little short- or medium-term impact on lead 
exposures. 

The most dramatic and creative effort at primary prevention was an eco- 
nomic disincentive against lead consumption, introduced as legislation by 
Congressman Ben Cardin of Maryland in 1993. Inspired by a suggestion 
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Table 3 Relationship between lead in gasoline and blood lead levels in the US population* 



Lead used in gasoline 

Median blood lead level 

Year 

(10‘ kg) 

(/ig/dL) 

1976 

186.47 

14.6 

1980 

51.59 

9.2 

1990 

0.47 

2.8 


‘Data from References 2, 53. 


from the Environmental Defense Fund (21), Cardin proposed a tax on lead, 
which would serve two purposes: Encourage product substitution by “leveling 
the playing field” for substitutes for lead, and provide fimds dedicated to the 
screening of children and abatement of past uses of lead, primarily lead paint in 
housing. This initiative failed in an anti-tax climate, but it represents something 
of a high-water mark in public health strategies in primary prevention. 

Screening is an important aspect of disease prevention, through the early 
identification of increased exposures and the prompt delivery of therapeutic 
interventions. Screening not only identifies individuals at risk, but it can also 
permit health authorities and others to identify sources of lead and reduce or 
remove them prior to other children being exposed. Screening for lead poison- 
ing involves the measurement of lead in blood, or an appropriately sensitive 
and specific biomarker of exposure. For lead, the biomarker of erythrocyte 
protoporphyrin (EP), a biological precursor in the cellular synthesis of heme, 
was widely used to diagnose lead exposures. This test was highly successful 
in facilitating screening because it was relatively cheap and, most importantly, 
with the hematofluorometer, results could be quickly obtained under clinic or 
field conditions (10, 52). Screening increased in the 1970s in many cities and 
slates until the early 1 980s, when screening decreased in the face of efforts to 
defund public health programs and roll lead screening into underfunded block 
grants to the states. 

When medical consensus caused CDC to lower the guideline defining toxic 
exposures to lead, first to 25 /rg/dl (1985) and then to 10 ug/dl (1991), the EP 
test was not sufficiently sensitive to identify children with PhBs in excess of 
these levels. The only recommended methods for screening involved careful 
collection of blood by venupuncture and lead analysis at a technically competent 
laboratory (1 1). The costs increased and logistics become more complicated. 


LEAD POISONING PREVENTION IN THE 1990S 

The successful regulation of certain lead sources substantially lowered average 
PbBs in children in the United States, from 1976 to 1991, As shown in Figure 3, 
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Figure 3 Distribution ofblood lead levels in US children (0 to 6 yrs) in 1976- 1980 fNHANES 11), 
as compared to that in 1988-1991 (NHANES III). Information on these studies is in Reference 8. 


the distribution of PbBs has shifted dramatically. The role of removing lead 
from gas in this reduction was clear from a trend analysis of the NHANES 
II data collected from 1976 to 1980, a period that coincided with the first 
major phasedown of lead in gasoline (Figure 4). While this change must be 
considered a public health success, this very success has reduced the momentum 
to fully eliminate lead poisoning. The disease has to a large extent become “re- 
ghettoized”. Now, children with PbB > 10 are much more likely to be black, 
live in large cities, and to be poor (Table 5) (8). For these children, lead paint is 
the overwhelming source of exposures (6 1 , 73), followed by exposure to highly 
contaminated urban soils that contribute to household dusts (39). However, as 
found by Ashley et al (3) when lead-painted surfaces are present in a home in 
poor condition, the paint source dominates in house dust and children’s blood. 

Lead-Based Paint 

Lead-based paint in housing has been the most significant source of high-level 
lead exposure for most of this century. As shown in Table 4, it is the most signifi- 
cant source of lead exposures for young children (1). Despite early recognition, 
restrictions on the use of lead in paint in the United States were successfully 
thwarted by the concerted actions of the organized lead industry (55). The city 
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Lttd Ut*d in QaiollM (100 metric tont/dty) 

Figui-e 4 Relationship between lead used in gasoline and median blood lead levels in the US 
population, 1976-1980 (NHANES II). Although not conducted for this reason, the NHANES 
survey of national lead exposures serendipitously coincided with the first major reductions in lead 
usage in gasoline. Later analyses, by Annest and others at CDC, demonstrated a high degree of 
correlation between monthly lead usage and blood lead (see Reference 2 for details). 

of Baltimore, under the inspired leadership of Huntington Williams, banned 
the use of lead-based paint in public housing, but it was not until 1977 that the 
US Consumer Product Safety Commission finally issued regulations limiting 
the concentration of lead in paint to 0.06%. This was a substantial reduc- 
tion, given that some earlier paint formulations were as much as 45% lead (by 
weight). However, the regulations covered only paints on indoor surfaces of 


Table 4 Estimated numbers of children aged less Ihan 7 years residing in unsound and 
lead-based paint US housing, by age and criteria of deterioration 


Category 

Construction date 

Lead-based paint homes number 

Children number 

Peeling paint 

Pre-1975 

1,972,000 

567,000 

Broken plaster 

Pre-1975 

1,594,000 

458,000 

Holes in walls 

Pre-1975 

2,602,000 

747,000 

Totals 

Pre4975 

6,168,000 

1,772,000 


Data from HUD. 
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Table 5 Blood lead levels in the US population, 1988-1991 


Population 

(years) 

Blood lead, 
Mcg/dL 

Percentage above 10 
/iCg/dL (geometric mean) 

Non-Hispanic white 

1-1 

3.5 

8.5 

3-5 

2.9 

3.7 

1-5 low economic status 


9.8 

high economic status 


4.3 

Black 

1-2 

6.3 

21.6 

3-5 

5.9 

20.0 

1-5 low economic status 


28,4 

high economic status 


5.8 

Mexican-American 

1-2 

4.2 

10.1 

3-5 

4.0 

6.8 

1-5 low economic status 


8.8 

high economic status 


« 


‘ Insufficient numbers for estimation. 
Data from Reference 8, 


residential structures, and articles (such ais toys and furniture) designed for use 
by children. 

More seriously, these steps could not deal with the lead paint that had been 
used in housing before the 1977 ban. As documented by the Department 
of Housing and Urban Development, over 40 million residential units in the 
United States are estimated to contain lead-painied surfaces because they were 
built before 1960 (Table 4). Of the 42 million units, some 26 million are in 
dilapidated condition (73), such that lead paint hazards are more likely to present 
risks of exposure through contamination of house dust and presence of paint 
flakes and chips (14, 15). Their distribution varies with the age of communities; 
Massachusetts, Illinois, New York, New Jersey, and Pennsylvania have millions 
of these units. 

Dealing with existing lead paint in housing has proved difficult because 
of the complex of issues related to poverty, housing, and disease prevention. 
Housing is an unusual vector of disease: It cannot simply be eradicated without 
provision of adequate substitutes. When the most dangerous housing is in poor 
neighborhoods, the social concerns of providing any affordable housing for the 
poor can come into conflict with preventing disease. 

The provision of lead-safe housing has been estimated to require major public 
or private investments, costing billions of dollars (73). Public policy, where it 
has existed (usually on the local level), has placed the burden on the private 
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sector to abate lead hazards when identified. For example, by 1980, Baltimore 
had developed, on paper, the most rigorous legal standards to require landlords 
to remove ail lead paint from homes where children with elevated PbB were 
found. 

Ford & Gilligan (22) were among the first to document the concerns about 
the practical problems with this policy. They evaluated the economic resources 
for private sector lead-paint abatement in Baltimore and concluded that land- 
lords could not recover the costs of adequate abatement from the revenues that 
could be generated fi-om rental housing in poor neighborhoods. They warned 
that existing laws, if fully enforced, would result in large-scale abandonment 
of housing and a loss of affordable housing for the poor. Their analysis was 
quickly taken up by property owners’ organizations to attack lead poisoning 
prevention programs in Baltimore, New York, and elsewhere. Their concerns 
were shared, for different motivations, by some advocates for housing and com- 
munity development, such as the Enterprise Foundation. The controversy was 
considerably heightened by the success of attorneys for some children and their 
families in winning large judgments and settlements against property owners. 

Despite applied research projects demonstrating that adequate hazard reduc- 
tion could be achieved for substantially lower costs than first estimated (19, 
20), the conflict remained. It came to a head after 1992, when legislation re- 
lated to lead-paint poisoning prevention had been passed through the efforts of 
Senators Alan Cranston of California and Paul Sarbanes of Maryland, among 
others. The Housing Act attacked the lead-paint problem in three ways. First, it 
closed a loophole in occupational health, whereby abatement and construction 
workers had not been covered by the 1978 OSHA lead standard (68). Second, 
it enlisted market forces to encourage abatement by requiring disclosure of 
lead paint in private real estate transactions. Third, it required HUD to develop 
policies for dealing with private and public rental housing, particularly for poor 
and low-income families. 

The real estate disclosure pro^am, finally implemented with regulations 
in 1996, may encourage source reduction, but these are likely to occur only in 
those housing markets able to sustain the investment required, that is, where the 
value of property can absorb an abatement “penalty” borne by the seller (either 
as reduced sale price or investment in abatement prior to sale). The success of 
disclosure and notification to prevent lead poisoning in children is unknown. 
Some anecdotal evidence suggests that it has worked in Boston in terms of 
increasing the amount of abatement (S Pollock, personal communication). But 
it has served to separate the lead poisoning problem of the affluent from that of 
the poor. The poor are more likely to be renters than owners, and their housing 
k in many cases insufficiently valued to absorb the costs of abatement without 
public subsidy. 
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The problem of low-income housing was given by HUD to a national task 
force to resolve. This task force, advisory to HUD, was chaired by two advo- 
cates, for housing (C Dolbeare) and lead poisoning prevention (D Ryan). The 
task force noted the scope of the lead-paint poisoning problem and attempted to 
resolve the conflicts between housing affordability and lead poisoning preven- 
tion by two strategies: lowering the standards for abating lead paint hazards, 
and inducing property owners to take preventive actions in return for insulation 
against civil litigation. The Task Force endorsed a policy of managing lead 
paint in place, rather than requiring actual paint removal or replacement of cer- 
tain structural elements, such as door and window frames [as recommended by 
Farfel & Chisolm (20)]. Property owners were to be immunized from litiga- 
tion if they undertook certain hazard-reduction steps. These recommendations 
were not universally endorsed. A minority of the Task Force (of which this 
author was one) rejected the concept of managing hazards in place without an 
enforceable system of ongoing inspection and maintenance, and they criticized 
the legal immunity proposal on the ground that it stripped away one of the few 
protections, admittedly imperfect, available to children at risk. 

Several states have adopted this strategy in law or practice. In Maryland, the 
state legislature passed a law that replaced Maryland’s rigorous program with 
the elements of the HUD Task Force recommendations. Similar proposals have 
been debated in other states. 

Lead-Contaminated Soil 

A similar retreat occurred in another arena where the expenses of remediation 
challenged the ambitious goal of source control. ATSDR had identified lead- 
contaminated soils as a significant source of lead exposures for children in the 
United States (1). Under the Superfirnd hazard ranking system, the presence 
of lead in soil tended to place a site on EPA’s National Priority List based 
on the assumption that lead in soil was significantly correlated with lead in 
children’s blood. Some 400 sites are on this list because of lead contamination. 
However, some studies of children living in communities with high soil-lead 
concentrations did not confirm this association. As reported by Freeman et al 
(23) and others (see 5), in some communities where the source of lead in soils 
was related to mining and smelting, no children with PbB above 10 /ug/dL were 
found. This was in contrast to experience in smelter and mining communities 
in El Paso, Texas, (33) or Port Pirie, Australia (the site of the world’s largest 
lead mining and smelting operations in the 1960s and 1970s) (38). Once again, 
an aspeet of social class seemed to play a role, as in the housing controversies. 
When the citizens of Aspen, Colorado, and Park City, Utah, two affluent resort 
towns in the Rockies, balked at being designated as Superfund sites, one citizen 
activist in Aspen remarked that “everyone knows that lead poisoning is a disease 
of poor Black kids and we don’t have any of [them] here” (New Yorker). 
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Studies on the association between urban soil lead and children’s PbB have 
been inconclusive. The EPA sponsored studies in Boston, Baltimore, and Cincin- 
nati to test the effects of reducing soil lead on children’s PbB. Relatively small 
changes were observed (76). The cost of cleaning up lead-contaminated sites 
was clearly very high, because only soil removal could work. If communities 
with high soil-lead levels were added to the Superfund list, it was not clear who 
would or could pay for abatement, since in some cases the source was not an 
industry but rather the use of lead in gasoline (39). 

In the face of this controversy, the primary prevention campaign again fal- 
tered. The EPA, charged with issuing standards for lead in soil, retreated to 
promulgating “advisory” levels that are not enforceable. They were not clearly 
based on any health-based concept of the potential contribution of lead in soils 
to lead in blood. 

Screening 

As the struggle for prevention failed in the case of housing and soils, another 
element in lead poisoning prevention also came under attack in the 1990s. 
Screening has always been an element of preventive health policy and early 
detection of lead exposure can prevent further exposures that result in irre- 
versible toxicity and can also assist in preventing exposures of other children 
to the same source of lead. But debate has arisen on the meaning of screening, 
and the reasonableness of measuring lead in blood of all children in the United 
States, 

In 1991, the Centers for Disease Control had recommended that all children 
under the age of two years should be screened at least once by blood lead 
testing (1 1). This recommendation was predicated on the NHANES II data 
that reported widespread prevalence of lead exposures (see Figure 3) and on 
the inadequacy of postexposurc treatment as a means of preventing or reversing 
toxicity. At the “high-water mark” of screening, in 1990, some 3 million 
children <6 years were tested. However, for screening to fulfill the purpose 
for prevention (primary or secondary) it must be part of an integrated program 
to identify and control the child’s lead sources, as well as to manage children 
with elevated levels. According to a survey conducted by the Association of 
State and Territorial Health Officials (ASTHO) in 1992, only 21 stales had 
implemented or were planning to implement the 1991 CDC guidelines. Less 
than half of the 48 states responding had a state system to monitor and follow 
up children with PbB > 10 ^tg/dL (4). In no state were all children with excess 
PhB followed up, with identification of exposure sources. 

Soon after the 1991 recommendation by CDC, criticisms were expressed 
from several sources. Physicians in California and Washington criticized the 
recommendation of universal blood lead screening as inflexible and unrespon- 
sive to local conditions (6). They argued that lead poisoning was unknown 
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Table 6 Blood lead levels in Maiyland, 1993® 


Area 

screened 

Number 

screened 

1-9 

Blood lead level, (%) 

10-14 15-19 20-24 

>25 

Baltimore City 

33,207 

21,188 

6804 

2933 

1264 

1028 



(63.8) 

(20.5) 

(8.8) 

(3.8) 

(3.1) 

Baltimore County^ 

4772 

4140 

441 

130 

36 

25 



(86.8) 

(9.2) 

(2.7) 

(0.8) 

(0.5) 

Howard County^ 

339 

311 

21 

4 

2 

1 



(91.4) 

(6.2) 

(1.2) 

(0.6) 

(0.2) 

Montgomery County‘s 

1794 

1728 

45 

15 

3 

3 



(96.3) 

(2.5) 

(0.8) 

(0.2) 

(0.2) 


®Dala from Maryland Lead Poisoning Prevenlion Program, Maryland Department of the Envirotiment, 
’’Counties immediately adjacent to Baltimore City. 

'Maryland’s most affluent county. 


in many communities where risk factors were low, and that in these situations 
requiring universal screening was an unacceptable waste of valuable health re- 
sources for the public and private sectors. While some localities reported that 
increased screening efforts were revealing a greater than expected prevalence 
of lead exposures in children (for instance, in Rhode Island), others found few 
if any children with PbB > 10 /rg/dL (many of these studies were published 
in Pediatrics ref 1994). The experience of Maryland in 1993 is illustrative. As 
shown in Table 6, the highest rates of elevated PbB were found in Baltimore 
City, with the lowest median family income in the state and the highest propor- 
tion of minority children, and a preponderance of housing built before 1950. 
In contrast, in Montgomery County, where median family income is among 
the highest in the United States, African American children comprise less than 
25% of the population, and where the housing stock is newer, the prevalence 
of PbB > 10 among screened children was less than 1%. 

In 1993, the American Academy of Pediatrics endorsed universal screening 
(16). But in 1994, Birt Harvey, of the Academy, recommended that universal 
screening should be abandoned (Harvey, 1994). He disputed the benefits of 
screening for children with PbB < 20 ;ig/dL; he pointed to the fact that in many 
communities few children were detected with PbB greater than 20 A^^g/dL; and 
he expressed concerns over laboratory resources and competence. 

Under pressure, CDC formally reopened its recommendation for universal 
screening in 1995. While it is probably true that in some communities, few if 
any children are exposed to lead such that their PbB exceeds 10 ^xg/dL (e.g. 7), 
the practical challenge is to develop a method that accurately defines these 
communities without missing children who are at risk of lead exposure (69). 
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Moreover, if such a strategy could be developed and validated, it is not clear that 
it would save health resources. The costs of blood lead screening can be calcu- 
lated; the costs of alternate screening methods have not been calculated. Yet the 
methods proposed by CDC recommend that health care providers administer 
and interpret an individualized questionnaire, in the context of information on 
those characteristics of the community that indicate the presence of lead expo- 
sure risks. This process has costs in terms of the time required by experienced 
health care providers and public health officials. 

Other criticisms have been leveled against screening of any type, notably by 
the US Preventive Services Task Force (74). This group concluded that: 

There is relatively little comnneing evidence Chat these Interventions [screening and en- 
vironmental or medical interventions based on screening] improve health, however. One 
issue is that most available studies in asymptomatic children evaluate the effects of various 
interventions on blood lead le\’els rather than on clinical outcomes. Second, blood lead 
levels typically decline with the passage of time. On average, blood lead levels in childhood 
decrease with age after peaking at about 2 years of age, even without intervention. Longitu- 
dinal studies of asyrapiorrutk children with elevated lead levels have shown reductions in 
blood lead levels after short- and long-term follow-up in the absence of any intervention, a 
result attributable at least in part to regression to the mean, random variation, and laboratory 
error. 

Evidence is not a\-ailabtc to demonstrate that universal screening for blood lead results in 
better clinical outcorrres than either screening targeted to high-risk persons or individual- 
ized testing in response to clinical suspicion. Several older studies reported that, compared 
to historical results from indn tduilized testing, intensive screening programs targeted to 
children in high-risk nei^borhoods reduced case fatality rates, mortality rates, and propor- 
tions of children detected with very high blood lead levels or who developed symptomatic 
lead poisoning. In the absence of concurrent controls, it is not clear whether the reported 
reductions in mortality and case fatality rates were due to screening, or to improvements in 
medical care over time. Reductions in mean lead levels may also have been due to secular 
trends, changes in screening tests, and to screening greater numbers of children, including 
many at low risk for sesere lead poisoning. Thus, the available evidence regarding the 
efficacy of screening programs is weak (74). 


BENEFITS OF PREVENTING LEAD POISONING 

Despite the controversies over how best to prevent lead poisoning, by screen- 
ing or environmental interventions, several analyses have shown significant 
net benefits to reducing lead exposures for children. The first of these studies 
was conducted by Schwartz and colleagues at EPA, in an analysis to support 
regulating lead in gasoline (See 46 for a review; 62 for the original analysis). 
As shown in Table 7, a correlation can be drawn between PbB and IQ in chil- 
dren based on several studies conducted worldwide (43). IQ is then correlated 
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Table 7 Meta-analysis, studies of the lead IQ relationship 


References* 

Year 

n 

Effect 

size 

Power to 
delect small 

effect 

p(l0 

Perino et al (29) 

1974 

80 

0.6 

0.2 

0.025 

Needleman et al (9) 

1979 

73 

0.35 

0.47 

0.015 

Yule et al (30) 

1981 

82 

0.573 

0.42 

0.021 

Winneke et al (31) 

1982 

26 

0.26 

0.18 

0.15 

Smith et al (19) 

1983 

185 

0.17 

0.7 

0.12 

Winneke et al (32) 

1983 

115 

0.351 

0.25 

0.4 

Harvey et al (21) 

1984 

48 


0 


Shapiro & Maracek (33) 

1984 

193 

0.46 

0.48 

0.025 

Lansdown ct al (34) 

1986 

162 

0.07 

0.48 

0.66 

Hansen et al (14) 

1987 

82 

0.5 

0.34 

0.0005 

Hawk et al (35) 

1986 

75 

0.64 

0.25 

0.0004 

Schroeder et al (36) 

1985 

104 

0.5 

0.33 

0.005 

Fulton et al (12) 

1987 

501 

0.4 

0.52 

0.003 

Hatzakis et al (13) 

1987 

509 

0.4 

0.52 

0.00065 






y^x = iw.ij 





P 

= 2.97 X 10-'- 


‘See Reference 45 for original citations. 


with academic attainment, from other literature, and academic anainmcnt with 
lifetime earnings. By these calculations, Schwartz (60) estimated that S6.9 
billion would accrue in benefits if children’s PbB were reduced by 1 ng/dL 
across the population. Salkever (59) updated these estimates to an increased 
benefits amount of $2.5 billion per birth cohort. [The other neurobehavioral 
sequelae of lead toxicity — antisocial behavior, school dropout, and criminal 
activity — ^were not monetized, largely because data are not sufficient to support 
a marginal analysis associating these outcomes in a dose-response relationship 
with incremental changes in PbB. Nevertheless, as pointed out by Needleman 
(43), these costs may be very great.] 

Large benefits of lowering blood lead levels were found for reducing lead 
in gasoline, lowering the drinking water standard, and abating lead paint in 
housing (60-62). However in some cases net benefits (exceeding costs) were 
found only when the value of reducing the effects of lead on blood pressure 
in adults and on materials damage (damage to spark plugs and plumbing) was 
added into the analyses. 

It is more difficult to calculate the benefits of screening. Briss and Schwartz 
have undertaken several cost benefit anal)^es, in order to support a rational 
basis for universal or less than universal screening (CDC, unpublished data). 
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A major problem lies in the controversy over the role of screening in lowering 
PbB as discussed above (74). 

CONCLUSIONS 

The prevention of lead poisoning achieved significant successes in the 1980s 
when new data on the nature and extent of lead toxicity provided political sup- 
port for major actions in primary prevention, notably the removal of lead fi’om 
gasoline. That action undoubtedly reduced environmental levels of lead in soils 
and air (42) and reduced blood lead levels across the US population (53). 

However, the campaign for prevention foundered by the end of the 1980s 
for three reasons. First, the “easy” tasks of source reduction (removing lead 
from gasoline, paint, and plumbing) were accomplished [it should be noted 
that the United States is the only country where these “easy” tasks have been 
achieved (50).] Second, these accomplishments reduced the urgency of lead 
poisoning as an issue among the US public as a whole, because lead poisoning, 
once again, was viewed as a disease affecting only a segment of the population, 
the disadvantaged of society, in a time when the political climate had largely 
turned against the “liberalism” of a welfare state. Third, the remaining sources 
of lead exposures for children — lead in housing and soil — presented daunting 
economic and technical costs for their solution. Even the relatively small costs 
of universal screening were considered by some to be too high. 

Yet lead poisoning has not disappeared. A disease with a national prevalence 
of 5 % cannot be considered to have been eradicated, as the US government once 
pledged to eradicate childhood lead poisoning (70). Moreover, a disease with 
such disparate prevalence, affecting those beset by other conditions of prejudice 
and disadvantage at a rate of over 25%, or five times more than advantaged 
children, should not be tolerated in a humane society. The costs of this neglect 
are bom by all, even if the risks of lead poisoning are not equally shared. 
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Key Words 

■ Abstract This review examines the first two studies conducted pursuant to a 
Congressional mandate that the U.S. Environmental Protection Agency analyze the 
effects of the Clean Air Aet on the “public health, economy, and the environment of 
the United States.” While these studies indicate that overall, the nation received good 
value for the resources it invested in improving air quality over the past three decades, 
we don’t know if even higher value could have been obtained by changing or eliminating 
certain potentially inefficient elements. The review focuses on the critical policy and 
technical choices made in the analyses, including the selection of the appropriate 
baseline and the level of disaggregation for the studies. It is proposed that a potential 
third analysis focus on potential new policies not yet mandated by law or regulation. 
It is also proposed that the next study fill in key information gaps, expand the benefit 
categories, and incorporate new research on topics such as mortality and morbidity 
benefits, cost uncertainties, and others. 


INTRODUCTION 

Since the Clean Air Act (CAA) was enacted in 1970, critics have repeatedly 
questioned whether the health and environmental benefits of air pollution control 
justify the costs incurred by industry, taxpayers, and consumers. Until Congress 
added Section 812 to the 1990 CAA Amendments, a requirement that the U.S. 
Environmental Protection Agency (EPA) conduct periodic, scientifically reviewed 
studies on the effects of the CAA on the “public health, economy, and the envi- 
romnent of the United States,” there was not a specific mandate for the Agency to 
compare the overall benefits of the CAA with the costs imposed on society. The 
resulting studies, The Benefits and Costs of the Clean Air Act: 1970 to 1990 (29) 
and The Benefits and Costs of the Clean Air Act, 1990 to 2010 (31), are widely 
seen as the “gold standard” of benefit-cost studies conducted by government, both 
in the United States and abroad. Under the auspices of the Advisory Committee 
for Clean Air Compliance Analysis (ACCACA), operating under the charter of 


0163 - 7525 / 02 / 05 1 0 - 0427 $ 1 4.00 


427 




Annu. Rev. Public. Health. 2002.23:427-448. Downloaded from \vww.annualreviews.( 


154 


428 KRUPNICK ■ MORGENSTERN 


00 

O 


o 


§■ 

S 


the EPA’s Science Advisory Board, both studies were scrutinized throughout the 
decade-long preparation by at least three expert committees of outside economists, 
air quality modelers, epidemiologists, and other health experts. They are prob- 
ably the most intensive and expensive benefit-cost analyses ever conducted at 
the EPA. 

Notwithstanding the many assumptions needed to address such a broad leg- 
islative mandate, the results of the analyses are clear: Overall, the nation has 
received high returns on its investment in improved air quality over the past three 
decades. For the early years benefits are estimated to exceed costs by an order 
of magnitude. Prospectively, benefits still exceed costs, although by a smaller 
margin. 

Particularly because the analyses were conducted at such a highly aggregate 
level, it is possible that the overall results reflect large gains from only a few 
programs within the CAA. Indeed, the overall finding of large net benefits may be 
masking ineffective or inefficient programs or regulations carried out under some 
titles of the CAA. In addition, many of the technical choices made in these studies 
involve important methodological issues about which experts differ. Understanding 
these issues is crucial to interpreting the overall findings. Currently, the EPA and 
one of its expert committees are meeting to consider the need for and the appropriate 
focus of a second prospective study. In contrast to the first two studies, the EPA has 
considerably greater latitude in defining the scope of such an analysis. Arguably, 
little would be gained by simply updating the existing studies with the same focus 
and methods used previously. 

The overall goal of this re\ ievi is to elucidate the results of the first two studies, 
including a critical examination of the policy and technical choices made in the 
analyses, and to highlight areas for improvement in a potential second prospective 
study. With this, the reader will develop some perspective on the practice and 
pitfalls of cost-benefit analysis as practiced by the government, as well as a better 
understanding of the net benefits to society of the CAA. 

This review emphasizes the disaggregation issue and areas where specific 
methodological choices could make a significant difference to the outcome. One 
additional attractive option discussed involves focusing the next study on a set of 
specific policy issues that have the potential to increase the net benefits of the CAA 
in the decades to come. 

The next section describes the basic methods and results of both the prospec- 
tive and the retrospective studies. Particular attention is paid to the baseline and 
aggregation issues. The third section considers alternative foci for a new prospec- 
tive study, including the option of focusing on the key policy challenges likely 
to be considered in coming years. The fourth section considers a wide range of 
technical and methodological issues including the need to fill in key information 
gaps, expand the benefit categories, and incorporate new research on topics such 
as mortality and morbidity benefits, cost uncertainties, and others. The last section 
draws overall conclusions and recommendations. 
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METHODS AND RESULTS OF RETROSPECTIVE 
AND PROSPECTIVE STUDIES 

The EPA’s retrospective and prospective sttidies were designed to examine a spe- 
cific set of policies that had been enacted by Congress since 1 970 and implemented 
by the EPA. The retrospective analysis focused on air quality policies and regula- 
tions put in place from 1 970- 1 990, and the prospective study addressed provisions 
added in the 1990 CAA Amendments. Both studies were conducted at a highly 
aggregate, economy-wide level. The retrospective study did not estimate either 
the benefits or the costs of individual regulations, pollutants, or any subcategories 
(e.g., stationary versus mobile sources) of the federal air pollution program. The 
prospective study estimated costs but not benefits by title of the CAA Amendments. 

From a policy perspective, an analysis of total benefits and total costs represents 
a very simple approach to a complex issue. Arguably, few propose abandoning 
all federal air pollution control. The more policy-relevant question concerns the 
benefits and costs of individual regulations and, even more relevant, the benefits and 
costs of marginal changes to individual regulations. The principal rationale'ofifered 
by the EPA for this highly aggregate analysis is that whereas costs can be reliably 
attributed to individual regulations or programs, the broad-scale methodology used 
for the benefits analysis precludes reliable estimation of the benefits by regulation 
or program, especially because some pollutants, e.g., nitrogen oxides (NO,), show 
up in multiple titles. This has been a highly controversial issue throughout these 
studies and remains so in the design of any new analysis. Indeed, the EPA’s own 
Regulatory Impact Analyses for Ozone and Particulate Mancr disaggregates the 
benefits of controlling these pollutants, although not by CAA Title. ^ 

In both the retrospective and the prospective studies the EPA analyzed air pol- 
lution programs by comparing specific policy and baseline scenarios. The retro- 
spective study contrasted a scenario reflecting historical economic and environ- 
mental conditions observed with the CAA in place with a hypothetical scenario 
that projects the economic and environmental conditions that would have existed 
assuming that the stringency and effectiveness of air pollution control technologies 
were frozen at their 1970 levels. In the prospective study all rules promulgated or 
expected to be promulgated pursuant to the 1990 CAA were contrasted to a sce- 
nario that essentially freezes federal, state, and local air pollution controls at the 
levels of stringency and effectiveness prevailing in 1990. Both studies hold con- 
stant the geographic distributions of populations and economic activities across 
the scenarios.^ 


*In addition, work by Smith & Ross (27) contains an analysis in which the Section 8 12 study 
results and other information is processed into a title by pollutant cost-benefit analysis. 
-Although the scenarios do reflect the basic trends in population and economic growth 
across the country over the relevant time periods, they do not allow for the possibility that 
people would respond to pollution by moving away from the dirtiest areas. 
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The frozen technology assumption — an obvious simplification — is central to 
the overall results. Arguably, in the absence of new federal regulation, one would 
expect to see some air pollution abatement activity, owing to state or local regu- 
lation or, possibly, on a voluntary basis. As Davies (9) has reported, nonfederal 
air pollution efforts date back to 1881 when the city of Chicago adopted an or- 
dinance that declared that “the emission of dense smoke from the smokestack 
of any boat or locomotive or from any chimney anywhere within the city shall 
be ... a public nuisance.” More recently, some states have imposed particularly 
stringent controls in some areas, e.g., California. If one assumed that state and 
local regulations would have been equivalent to federal regulations, then a benefit- 
cost analysis of the federal CAA would be a meaningless exercise: Both be- 
nefits and costs would equal zero. For both studies, the EPA and the outside 
experts wrestled with the possibility of developing more realistic baseline scena- 
rios. In the end they decided that any attempt to predict how states’ and local- 
ities’ regulations or voluntary efforts would have differed from the CAA is too 
speculative. 

Each of the two (aggregate) scenarios is evaluated by a sequence of economic, 
emissions, air quality, physical effect, economic valuation, and uncertainty models 
to measure the differences between the scenarios in economic, human health, and 
environmental outcomes. Both studies examine the benefits and costs of reducing 
volatile oi^anic compounds, nitrogen oxides (NOx), sulphur dioxide (SOj), car- 
bon monoxide (CO), coarse particulate matter (PMjo), and fine particulate matter 
(PMa.s).^ The retrospective analysis assessed the effect of CAA provisions gov- 
erning lead in the environment. However, because the 1990 Amendments do not 
include new provisions for the control of lead, it is not considered in the prospective 
analysis. 

Although both studies attempt broad coverage, there arc some notable omis- 
sions, largely because of data or modeling limitations. Emissions of hazardous air 
pollutants are not extensively considered in either study."* Recent revisions to the 
particulate matter and ozone ambient standards are also omitted from the prospec- 
tive study, although the EPA analysis indicates that because of similarities in the 
baseline assumptions, the benefits and costs reported in the Regulatory Impact 


^Although the incremental effects of the CAA Amendments on primary particulate matter 
emissions are relatively small, particulate matter in the atmosphere is comprised of both 
directly emitted primary particles and particles that form in the atmosphere through sec- 
ondary processes as a result of emissions of SO 2 , NO„ and organic compounds. These 
particulate matter species, formed by the conversion of gaseous pollutants emissions, are 
referred to collectively as secondary particulate matter. Because the CAA, especially the 
1990 Amendments, achieved substantial reductions in these gaseous precursor emissions, 
it has a much larger effect on PMj® and PM 2,5 than might be apparent if only the changes 
in directly emitted particles are considered. 

"•Some pilot analyses of hazardous air pollutants were conducted but it was determined that 
the poor quality of the available information precluded comprehensive quantification of the 
effects. 
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Analyses for these pollutants can be considered incremental to the results of the 
prospective analysis. Estimates for that part of the CAA Amendments regarding 
stratospheric ozone depletion (Title VI) are developed in the prospective study but 
they are not fully integrated into the main analysis. 

Emissions estimates reflect the expected growth in population, transportation, 
electric power generation, and other economic activity over the relevant time pe- 
riods. Different estimation procedures arc used for stationary, mobile, and area 
sources, although the benefit and cost estimates are not disaggregated in that man- 
ner. Costs are estimated as increases in expenditures by different entities to meet the 
additional control requirements of the CAA, including operation and maintenance 
expenditures plus amortized capital costs (i.e., depreciation plus interest costs as- 
sociated with the existing capital stock). Changes in employment and prices as well 
as impacts that might be experienced among customers of the firms that must incur 
these costs were partially examined in the retrospective analysis but omitted in the 
prospective study. In limiting consideration of these so-called general equilibrium 
effects, the EPA reports effectively preclude analysis of the economy-wide costs 
of imposing additional environmental regulations in the context of existing labor 
and other (disiortionary) taxes.^ 

Although the air quality modeling efforts focused on the full range of pollu- 
tants, both studies found that the majority of the total benefits are attributable 
to changes in paniculate matter concentrations. Consistent with current scientific 
understandings, neither the specific source categories nor the chemical composi- 
tion of the panicles were considered. Thus, secondary particles formed from SO 2 , 
NOs, and volatile organic compounds were all treated (uniformly) as fine particles. 
The retrospective study found significant benefits associated with the reductions in 
lead — principally the phase-down of lead in gasoline. The monetary benefits of air 
quality improvement include reduced incidence of premature mortality and other 
human health efifecis, as well as improvements in visibility and avoided damage 
to agricultural crops. Despite efforts to characterize the impacts of air pollution 
on natural systems, the inability to quantify and/or place a monetary value on the 
damages precluded the development of benefits estimates for ecosystem impacts 
(except for a supplementary calculation for avoided costs of nitrate reductions; see 
below). A similar story applies to potential carcinogenic and certain other health 
effects associated with criteria pollutants. 

The monetary benefits reflect interpretations of the available science and eco- 
nomic literature made by the EPA in consultation with its outside experts. As a 
form of sensitivity analysis, a number of alternative interpretations of the literature 
were also examined. The quantitatively most important concern the valuation of 
premature mortality. In both the retrospective and prospective analyses, the EPA 
developed an alternative scenario based on the loss of life-years approach to reflect 


•‘'Costs for meeting Title IV tlirough the SO 2 trading program were estimated by a model 
that allocates emissions reductions cost-effectively in a context of responding to market 
signals in the electric power and tradable allowance markets. 
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the greater susceptibility of older individuals to air pollution-induced mortality. 
In both studies this scenario yielded significantly lower benefits. The prospective 
study also examined alternative assumptions about the incidence of mortality and 
the incidence and valuation of chronic bronchitis, as well as certain other effects. 
Sensitivity analysis was used to examine alternative behavioral responses to strato- 
spheric ozone depletion (Title VI), such as remaining indoors or increasing use of 
sunscreen or hats. 

Table 1 displays the present value of monetary benefits of the CAA by endpoint 
category, along with comparisons to estimated costs, for 1970-1990. An array of 
benefit estimates is presented, reflecting the underlying uncertainties. Overall, the 
present value of total benefits is estimated to range from S3.5 trillion (5th percentile 
of the underlying probability distribution) to S56.3 trillion (95th percentile) over 
the 20-year period. Comparing the mean benefit estimate (S22.2 trillion) with the 
present value point estimate of costs (S.5 trillion), benefits are seen to dwarf costs 


TABLE 1 Present value of 1970-1990 costs and monetized benefits by endpoint category for 
population of continental United States (billions of 1990 dollars, discounted to 1990 at 5 %) 
[from Reference (29)] 

Present value 


5th 9Sth 


Endpoint 

PoUutant(s)* 

pertenlile 

Mean 

percentile 

Mortality 

PM 

S2J69 

$16,632 

$40,597 

Mortality 

Pb 

S121 

$1,339 

$3,910 

Chronic bronchitis 

PM 

S409 

$3,313 

$10,401 

IQ (lost IQ pts. -t- 
children w/IQ <70) 

Pb 

S27I 

$399 

$551 

Hypertension 

Pb 

S77 

$98 

$120 

Hospital admissions 

PM, 03, Pb, CO 

S27 

$57 

$120 

Respiratoiy-related symptoms, 
restricted activity, 
decreased productivity 

PM, 03, NOj. SOi 

SI23 

SI82 

$261 

Soiling damage 

PM 

S6 

$74 

$192 

Visibility 

particulates 

S38 

$54 

$71 

Agriculture (net surplus) 

03 

Sll 

$23 

$35 

Total benefits 


S3,452 

$22,171 

$56,258 

Total costs 



$523 


Net benefits (total 



$21,648 



benefits — total costs) 


“PM, particulale matter; Pb, lead. 
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TABLE 2 Alternative mortality benefits’ mean estimates for 1970-1990 (in 
trillions of 1990 dollars, discounted at 5%) [from Reference (29)] 


Benefit estimation method 

Benefits particulate matter 

Total 

Statistical life method ($4.8 million/life) 

16.6 

18.0 

Life-years lost method (S293,000/year) 

9.1 

10.1 




by a factor of 42:1.^ Net benefits amount to $21.6 trillion. Note that according to 
the mean estimates particulate matter mortality is estimated to account for 75% of 
total benefits, followed by chronic bronchitis (15%) and lead mortality (6%). The 
remaining eight categories comprise the residual 4% of benefits. 

Table 2 presents alternative particulate matter mortality benefits when the life- 
years lost method as opposed to the statistical life approach is used in the cal- 
culations. Although the latter approach has historically been used by the EPA in 
conducting its benefit-cost analyses, some short-term particulate matter exposure 
studies suggest that a disproportionate share of particulate matter-related prema- 
ture mortality occurs among persons 65 years of age or older. Thus, at the urging 
of the outside review committee, the EPA combined standard life expectancy ta- 
bles with the limited available data on age-specific incidence to develop (crude) 
approximations of the number of life-years lost by those who die prematurely as 
a result of exposure to particulate matter. These were presented as alternative es- 
timates. As shown in Table 2, the particulate matter mortality benefits, the largest 
single benefit category overall, fall by 45% when this alternative valuation method 
is used. Although use of the life-years lost method does not change the basic con- 
clusion of the retrospective study that the monetized benefits greatly exceed costs 
over the period 1970-1990, the magnitude of the change demonstrates the impor- 
tance of answering the question of valuation with greater clarity. As discussed in 
Updating The Information, below, the life-years lost method is but one of several 
current issues in the valuation literature. 

Table 3, drawn from the prospective study, summarizes the central estimates 
on a present value basis of the costs and benefits of the Clean Air Act (CAA) 
for 1990-2010. About 90% of the benefits are associated with avoided mortality, 
a slightly higher proportion than in the retrospective study. The remainder of the 
benefits are associated with avoided morbidity and ecological and welfare benefits. 
On the cost side, the prospective analysis finds that Title I [National Ambient 
Air Quality Standards (NAAQS)] accounts for almost half of the total cost of 
the first five titles. Title II (mobile sources) accounts for another third, with the 
balance distributed among Title III (toxic emissions). Title IV (SO 2 and NO^ from 
power plants), and Title V (permitting). Because of the long-term nature of the 


^Note that the costs were treated as if they were certain, when in fact there is much uncertainty 
about such costs (see Updating The Information, below). 
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TABLE 3 Summary of quantified primary central estimate benefits and 
costs of Clean Air Act amendments, 1990-2010 (estimates in millions of 
1990 dollars) [from Reference (3I)]“ 




Annual estimates 

Cost or benefit category 

2000 

2010 

Present value 

Costs: 

Title I 

$8,600 

$14,500 

$85,000 

Title II 

$7,400 

$9,000 

$65,000 

Title III 

$780 

$840 

$6,600 

Title IV 

$2,300 

$2,000 

$18,000 

Title V 

S300 

$300 

$2,500 

Total costs, Title I-V 

519,000 

$27,000 

$180,000 

Title VI 

SI, 400* 


$27,000* 

Monetized benefits: 

Avoided mortality 

$63,000 

$100,000 

$610,000 

Avoided morbidity 

$5,100 

$7,900 

$49,000 

Ecological and welfare effects 

$3,000 

$4,800 

$29,000 

Total benefits, Title I-V 

$71,000 

$110,000 

$690,000 

Stratospheric ozone 

$25,000* 


$530,000* 


“Annual estimates for Title VI stratospheric ozone protection provisions are annualized equivalents 
of the net present value of costs from 1990 to 2075 (for costs) or 1990 to 2165 (for benefits). The 
difference in time scales for costs and benefits reflects the peraistence of ozone depleting substances 
in the atmosphere, the slow processes of ozone formation and depletion, and tl>e accumulation of 
physical effects in response to elevated UV-b radiation levels. 


benefits of Title VI (stratospheric ozone), its results are not fully integrated into the 
overall findings. However, the present value benefits of this title exceed costs by a 
factor of twenty. Overall, while the monetized benefits of the CAA Amendments 
over the period 1990-2010 still exceed the costs, the ratio of benefits to costs 
(about 4: 1 for Titles I-V) is considerably lower than in the retrospective analysis, 
suggesting that the “truly low hanging fruit” may have been picked in the early 
years.^ 

Overall, as the EPA has noted in the prospective study, the conclusion of the 
Section 812 analysis is clear: 

While alternative choices for data, models, modeling assumptions, and valu- 
ation paradigms may yield results outside the range projected in our primary 
analysis, we believe based on the magnitude of the difference between the 
estimated benefits and costs that it is unlikely that eliminating uncertainties or 
adopting reasonable alternative assumptions would change the fundamental 


^In one of the scenarios presented in the prospective study (low benefits), costs actually 
exceed benefits by $1 billion per year. 
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conclusion of . . . [the] study: the Clean Air Act(s’) . . . total benefits to society 
exceed its costs. (3 1 , page v) 

Although the findings of these studies have been discussed in the media and, 
to some extent, on Capitol Hill, it is feir to say they have played only a modest 
role in the policymaking process. In our judgment, the EPA needs to disaggregate 
the benefits and costs into those applicable to specific titles, sectors, or regulations 
in order to have a significant impact on policy decisions. Although the EPA did 
not do this in the Section 812 study, others have done so for specific titles of the 
CAA, taking the EPA’s aggregate benefit estimate (and cost estimates by title) as 
given (27). Extensive analysis has also been conducted on the electricity generation 
sector (Title IV) alone (7, 5). In addition, the EPA was able to develop separate 
benefit estimates for their new ozone and fine particulate National Ambient Air 
Quality Standards (30). The findings from these studies are presented in Table 4. 
This table shows that some titles deliver more net benefits than others and that the 
new fine particulate NAAQS is likely to be a much better buy for society than the 
new ambient ozone standard. 


TABLE 4 Summary of cost-benefit studies of the 1 990 Clean Air Act 
Amendments for 2010 (estimates in millions of 1990 dollars) 


Study 

Benefits 

Costs 

Title IV 

Burtraw et al. (5)* 

$25,000 

$800 

Chestnut (7) 

$35,277 

NA 

New NAAQS (30)*’ 

Ozone (8-h), partial attainment 

$400-$2,100 

$1,100 

Ozone (8-h), full attainment 

$l,500-$8,500 

$9,600 

Fine particulates, partial attainment 

$19,000-$104,000 

$8,600 

Fine particulates, full attainment 

$20,000-$! 10,000 

$37,000 

Clean Air Act Amendments (27)® 

Title I 

$26,564 

$14,500 

Title II 

$14,968 

$9,000 

Title III 

$1,925 

$840 

Title IV 

$69,297 

$2,000 


^Whiic this estimate is specific to Ute easlem United States, these benefits are expected to 
account for 98% of total U.S- benefits. 

^Partial anainment costs are incremental to partial attainment of current standards and 
reflect partial attainment of promulgated standards. The EPA estimates 17 potential residual 
nonattainmenl areas for ozone and 30 potential residua! nonattainment areas for fine particulates 
as of 2010. Full attainment costs, however, are incremental to full attainment of current 
standards. 

‘^Total CAA Amendments benefit estimate ($li0 billion; see Table 2) and cost estimates by 
title (see Table 2) are from Reference (31). 
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DESIGN OF A NEW PROSPECTIVE STUDY 

The scope of the first two studies was clearly established by Congress based on 
policies and regulations implemented prior to 1990 or, in the case of the prospec- 
tive analysis, based on the new provisions of the 1990 CAA Amendments. In 
contrast, the framework for a second prospective study is less well defined. There 
have been no revisions to the Act since 1990. Thus, there is not a clear set of Con- 
gressionally mandated policies to examine. We believe this absence of specific 
Congressional direction creates an opportunity for the EPA to establish its own 
agenda for the second prospective study that enhances the opportunity to further 
incorporate economic considerations in future policies. 

How should the EPA best approach a new Section 812 study? Some might 
argue for a simple updating of the first prospective study, with the addition of new 
data and research results. In our judgment, it would be unwise to proceed with 
such an approach, because the new information generated is unlikely to justify the 
considerable resource costs involved. Further, the results of such a study would 
likely have only marginal policy impacts, as most of the regulations considered 
in such an analysis would have been implemented by the time the study was 
completed. We believe the new study should focus on critical policy issues likely 
to be considered by the EPA over the next decade. Specifically, we propose that the 
EPA use a new Section 8 1 2 study as a vehicle for considering how it can improve the 
net benefits of the CAA in the coming decades. The net benefits framework assures 
that potential opportunities to increase total benefits as well as those that would 
decrease total costs are examined. Because the study would likely be completed 
prior to Congressional and EPA action on these new policies (or alterations of 
existing policies), there would be a clear opportunity for the analysis to influence 
policy decisions. If conducted in a timely way, such a study would represent a 
departure fi’om the first two Section 812 studies, which were conducted after the 
bulk of the policy decisions were made. Indeed, using the new prospective study 
in this planning function would show Congress that the EPA is serious about using 
benefit-cost analysis to inform its regulatory agenda and decision-making. 

In contrast to the first two studies, in which the focus was largely on aggre- 
gate analyses, a new study aimed at improving the design and implementation 
of the CAA requires an understanding of how specific elements of the Act are 
performing. To make decisions on the efficacy and efficiency of specific elements 
of the CAA requires costs and benefits disaggregated in a useful manner. The dis- 
aggregations could include the stringency of air quality standards for ozone and 
particulate matter (Title I); their implementation through the State Implementation 
Plan process, including various requirements for mobile source controls (Title II); 
the provisions for SO 2 allowance trading and reductions in NO^ from power plants 
(Title IV); and provisions for control of toxic emissions (Title III) and permitting 
(Title V). 

Overall, we believe that the new Section 8 12 study should also focus onpotential 
new policies not yet part of EPA policy or regulation. The so-called multi-pollutant 
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bills, which focus on SO 2 , NOx, mercury, and possibly CO 2 , would be strong 
candidates for consideration. It would be particularly useful to consider the benefits 
and costs of alternative stringency levels for each of the pollutants, as well as the 
interaction among them. For example, how do the marginal costs (and benefits) of 
controlling CO 2 vary with alternative stringency levels of SOx, NOx, and mercury? 
The tradeoffs being contemplated between tightening down on these pollutants and 
loosening or eliminating New Source Review requirements would be particularly 
illuminating. It would also be useful to examine other initiatives that might be 
considered to meet new or anticipated NAAQS requirements, e.g., new regulations 
whose primary goal is to reduce ground-level ozone or those primarily designed to 
reduce fine particles. When doing this kind of analysis it will be important to note 
how some regulations (e.g., those dealing with nitrogen oxides and volatile organic 
compounds in particular) provide benefits in terms of tropospheric ozone, as well 
as fine particle reductions. Other possible areas of interest include new policies to 
reduce emissions of hazardous air pollutants, possibly including so-called residual 
risk standards. For all these policies it would be appropriate to analyze a number 
of different scenarios that attempted to encompass the bounds of possible new 
policies. 

Consistent with the goal of illuminating issues likely to have particular policy 
relevance over the next decade, the most important policies to analyze would be 
those with potentially high costs, whose benefits are uncertain. Examples would be 
the currently mandated bans on episodic controls currently in the CAA, the current 
New Source Review process, the Inspection and Maintenance Program, and even 
the State Implementation Plan conformity process. These represent existing parts 
of the Act in which alternative approaches may yield significant environmental 
benefits relative to the costs incurred. It is also possible that selected regulations 
mandated over the past several years but not yet implemented could be reconsidered 
in the Section 812 context. 

In general, the appropriate level of aggregation depends on the use to which the 
analysis is to be put. The EPA’s review committee has previously uiged the EPA 
to pursue a title-by-title approach (ACCACA 1999). To a limited extent the EPA 
has done so in the first prospective analysis by estimating the costs of the 1990 
amendments on a tiile-by-title basis. However, because of the difficulty of uniquely 
attributing individual pollutants to specific titles, it is difficult to break out benefits 
by title. In the case of SO 2 and NOx reductions from power plants (Title IV), this is 
relatively straightforward. However, NOx controls show up in three separate titles. 
For future analyses it may be more appropriate from both a methodological and a 
policy standpoint to distinguish regulations on a broad sectoral basis, e.g., station- 
ary, mobile, and area sources. To the extent feasible, it would also be desirable to 
seek finer distinctions within these categories, e.g., regulations on electric utilities, 
petroleum refineries, and other large sources. Individual as opposed to groups of 
policies should be examined to provide the maximum amount of information to 
decision makers. Benefits and costs computed by sector can indicate the relative 
efficiency of controls or other emissions management options aimed at different 
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pollution source categories. For example, in such a framework it would be possible 
to compare the net benefits of ozone strategies focusing on emission reductions 
from stationary sources (e.g., NO* controls on the electric power industry) with 
motor vehicle strategies (e.g., enhanced inspection and maintenance programs). 


UPDATING THE INFORMATION 

The EPA’s prospective study was published in 1999, but most of the analyses were 
completed in 1998. Since then, progress has been made in a number of research 
areas, including the valuation of health effects and cost analysis. In this section 
we focus on the new economic research that we believe could and should be 
incorporated into a second prospective study, and by extension, any cost-benefit 
analysis of this type of pollution. At the outset, however, we note new developments 
in the areas of air pollution modeling and in the epidemiological studies used to 
estimate health effects. 

The first prospective study relied on one air quality model to convert SO 2 and 
NOx emissions into particulate air pollution and another to convert NOx and volatile 
organic compound emissions into ozone. Unfortunately, this approach did not allow 
the'air chemistry underlying these conversions to be addressed in a unified fashion. 
The newest generation of air quality models addresses both secondary particulate 
and ozone formation in a single, comprehensive and internally consistent model. 
Examples of these models include EPA’s MODELS3 (32) and the Georgia Tech 
Model (37). These models arc very complex and expensive to set up, so the new 
analysis probably cannot depend entirely on them. Nevertheless, there is for the 
first time a capability to examine how the use of a unified air quality model alters 
the benefit estimates relative to the framework previously used. 

Recent epidemiological evidence has led to some new questions and an under- 
standing of the effects of particulate pollution on mortality in adults. Similarly, 
there is growing evidence of the effect of elevated concentrations of particulate 
matter on infants — a category that was excluded from both the retrospective and 
the pro.spective studies. In the Health Effects Institute-led re-analysis of the long- 
term study by Pope et al. (24), which was the primary study used by the EPA to 
estimate mortality effects in their prospective analysis, Krewski et al. (13) clearly 
reaffirmed the qualitative relationships but found several interesting anomalies.® 
Other recent studies, both long-term and short-term, came to similar conclusions. 

Considering the short-term studies, Samet et al. (25), in their 20 and 90 cities 
studies, found an aggregate coarse particulate matter (PMio) effect of about half of 
that estimated by earlier short-term studies. They also found evidence of region- 
specific variation in mortality effects, most notably in the northeastern United 


^Long-term studies involve followinga cohort ofindividuals in manycities for years, relating 
their pollution exposures to the probability of death. Short-term studies involve examining 
how daily death rates in a given city vary with daily pollution exposures. 
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States, where mortality effects were estimated to be twice as high as those for 
the 90 cities overall. Zanobetti & Schwartz (38), in examining potential socioe- 
conomic modifiers of PMio effects, found the mortality effect for females to be 
approximately one third larger than that for males. Interestingly, adding sulfates 
(SO 4 ) and some heavy metals to such models significantly lowers the fine partic- 
ulate matter (PM 2 . 5 ) effect (4), whereas adding gaseous pollutants generally does 
not affect the particulate matter coefficients (25). Wichmann et al. (35) finds that 
the ultra-fine particles (PMj or lower) are a significant predictor of mortality, until 
other pollutants are added. 

With respect to the long-term studies, Krewski et al. (13) found slightly larger 
effects of PM 2.5 on mortality than that found by Pope et al. (24), but these effects 
fall dramatically and even become insignificant under some specifications, while 
SO 2 effects become large and significant. In addition, the mortality effects of fine 
particles (PMi.s) varied with education level, the estimated effect being higher for 
individuals without a high school education than for those with higher levels of 
education. Furthermore, Abbey et al. ( 1 ) found no relationship of PM 2.5 for females, 
and Lipfert et al. ( 1 8) in a major study of veterans’ mortality and pollution, found 
no PM effects, whereas they found ozone and NO 2 to be significant predictors of 
mortality. 

At the same time that these new analyses are introducing new questions about 
mortality effects in adults, several studies around the world have strengthened the 
case for infant mortality being caused by exposure to particulates. Woodruff et al. 
(36) found mortality for respiratory causes and sudden infant death syndrome to 
be positively associated with high PMio exposure for normal birth weight infants 
but no significant relationship for low birth weight infants. Lipfert et al. (19) 
replicated the findings of Woodruff et al., though they did not observe differences 
among categories of birth weights. Also, Bobak & Leon (3), who conducted a 
matched population-based case-control study covering all births registered in the 
Czech Republic from 1989 to 1991, found that only particulate matter showed a 
consistent association with death when all pollutants were entered in one model; 
these effects were strongest in the postneonaial (rather than the neonatal) period 
and were specific for respiratory causes. 


Filling in Valuation Gaps 

A number of key linkages between pollution and endpoints that people value were 
missing from the prospective study. These include visibility, ecosystem damage, 
negative benefits (i.c., increased risks of melanomas) to ozone control from added 
exposure to ultraviolet radiation (UV-B), and effects of toxic exposure. Many of 
these benefit categories have been examined in recently published literature reviews 
[e.g., Cropper ( 8 )]. The ACCACA (2) questioned the quality of all the visibility 
benefits studies, either because of methodological problems or because they had not 
been peer reviewed. Although the academic interest in valuing visibility benefits 
may have waned, perhaps from a lack of recent research support, the available 
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studies suggest that such benefits could comprise a significant fraction of total 
benefits. In addition, the methods for eliciting values for visibility in a recreational 
context and in the western United States appear to be reasonably reliable. So few 
studies are available to set a value for urban visibility benefits that this area must 
be considered more speculative. 

Ecosystem damages were largely ignored in both the retrospective and the 
prospective EPA studies, for the good reasons that there are no studies suitable for 
estimating physical ecosystem level benefits from air pollution reduction at the 
broad regional level (except in very specialized areas, such as the Adirondacks). 
Also, there are no studies providing a firm basis for valuing such improvements at 
this broad level, even in places where physical damage might be estimated. 

The prospective study attempts to capture some ecosystem damage by count- 
ing foregone cleanup costs as a reasonable proxy for losses from nitrification of 
Chesapeake Bay and other waters associated with NOx emissions. The expert 
review committee was adamant that this approach was an unreliable estimate of 
benefits because, except in special cases, there is no necessary relationship between 
foregone cleanup costs and the benefits of cleanup. 

Finally, both studies were silent on the possible increase in skin cancers as- 
sociated with increased UV-B radiation, itself associated with ozone reductions. 
Lutter & Wolz (20) and others have estimated these effects and find them to be 
nontrivial. The EPA ignored these effects, as it did in the regulatory impact analysis 
for its recently promulgated new ozone standard. This action was rejected by the 
Supreme Court (33) and the EPA agreed to consider this effect. Thus, there is no 
reason why future cost-benefit analyses of the CAA should omit what could be an 
important perverse effect of reducing ozone air pollution. 


Using Better Methods/Newer Studies 

The most important valuation number in the report is the value of a statistical 
life (VSL), which is the average willingness to pay for a given (small) reduction 
in risk of death divided by that risk reduction. The product of the VSL and the 
expected number of deaths avoided by the CAA yields an estimate of benefits that 
represents almost 90% of all benefits calculated in the Section 812 studies. The 
EPA’s VSL estimate of S6 million was based on a review of 26 of the studies in 
the literature — 21 labor market studies and 5 contingent valuation studies. The 
analytical methods applied to this review were ad hoc at best, and the reason for 
choosing the 26 studies was never entirely clear. 

We believe that improvements in these estimates could come from two distinct 
elements of the health valuation literature. The first is a systematic evaluation of 
this literature. Miozek & Taylor (22) have performed a meta-analysis of 38 studies 
contributing 203 VSL estimates. They found that the EPA’s best estimate for the 
VSL ($6 million in 1 998 dollars) is three times too large, i.e., Mrozek & Taylor’s 
best estimate is S2 million, owing to a number of factors. The most important of 
these is a false attribution of wage rate differentials to mortality rate differences. 
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when in fact much of this variation is due to interindustry differences in wage rates 
that occur for other reasons — a point first raised by Leigh (17). 

Just as importantly, there are some new studies in the mortality risk valua- 
tion literature [e.g., Hammitt & Graham (11), Krupnick et al. (15), Strand (28), 
Johannesson & Johansson (16)] that are specifically designed to reflect the mortal- 
ity risks associated with air pollution, but using contingent valuation and conjoint 
analysis approaches. Much of this literature also suggests that the EPA’s S6 mil- 
lion estimate for the VSL is too high, with the appropriate adjustment being quite 
uncertain, as this literature needs to mature. Nevertheless, some of the literature 
suggests that the consideration of dread (e.g., with cancer-causing pollution ex- 
posures) and altruism (flie willingness to pay for health improvements for others) 
could significantly increase the VSL. 

The second most important valuation number in these studies is the value of 
a case of chronic disease. This is the analogue of the VSL, but for the reduced 
risk of getting chronic lung or heart disease. The estimate used in the EPA reports 
(5266,000) was derived from two small (300 person) pilot conjoint analysis studies 
(34, 14) that scarcely can support the weight placed upon them. Conjoint analy- 
ses and other stated preference techniques have evolved since these studies were 
done. Unfortunately, no new numbers are available for improving on this estimate. 
Clearly, this is an important area for future research, if only to do the same studies 
with a larger sample size. 

Estimating Costs 

The EPA has a long history of conducting its analyses of the costs of regulations 
based on engineering abatement cost estimates that are generally “on the shelf.” 
This approach has the advantage of being transparent and easy to defend. One 
chooses technologies to abate a pollutant from a set of agreed upon technologies, 
the most cost-effective first. 

Unfortunately, this approach is likely to lead to overestimation of control costs 
for two reasons. First, it does not account for future technological advances that 
may bring down such costs (or raise effectiveness). This is a particularly important 
failing when time horizons are long. Second, it often ignores options for reducing 
costs that are not classified as engineering approaches or do not show up in the 
approved list of technologies. This applies, in particular, to market or economic 
incentive approaches to pollution control, which may involve changes in input mix, 
process changes, product redesign, or others that respond to the need to minimize 
abatement costs. Recent work by Harrington et al. (12) has found empirical support 
for the notion that ex ante cost estimates tend to overstate true costs. In examining 
ex post infonnation, Morgenstem et al. (21) found that reported enviromnental 
expenditures are generally not overstated, although considerable variation exists 
at the plant level. 

The EPA’s prospective study, while generally ignoring the cost consequences 
of incentive approaches, did consider them in the one place where they would be 
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particularly important, namely for reducing SO 2 from power plants. Title IV of the 
CAA Amendments of 1990 set up an SO 2 allowance trading system for utilities to 
substantially reduce their SO 2 emissioiLS through a trading program. The benefits 
of such a program (forecast out to 2010 estimated from the Integrated Planning 
Model for the EPA), which accounted for fuel switching, use of lower sulfur 
coal, new and retiring plants, demand reductions in reaction to higher electricity 
prices and generation load reallocations as ways of meeting the standard. Several 
other studies (5, 6, 10) also estimated costs and generally corroborate the EPA’s 
estimates. 

An omission in the report, albeit one whose inclusion would significantly raise 
costs, is the cost to society of imposing environmental regulations in the context 
of the existing tax structure — the so-called tax interaction effect (23). The tax 
interaction effect refers to the economy-wide economic losses associated with the 
imposition of additional environmental (or any other) regulation — losses that tend 
to be exacerbated by existing labor and other taxes. Aggregate losses are potentially 
quite large because there are so many people affected.^ The tax interaction effect 
was extensively discussed by the expert review committee and the EPA and is 
mentioned in the prospective study, but it is not incorporated quantitatively. Clearly, 
future prospective studies could and should take this effect into account. 

Inclusion of Other CBA Studies 

Although the EPA studies are quite thorough in referencing the literature underly- 
ing key elements of the analysis, they generally omit references to other integrating 
studies similar in nature to the Section 812 reports. This is in marked contrast to 
benefit-cost studies in the academic literamrc, in which one can often find com- 
mentary on estimates from similar studies and an attempt at reconciliation of any 
differences that arise [e.g., Burtraw et al. (5)]. One good example pertains to the 
EPA’s Regulatory Impact Analysis of the Proposed Ozone Standard (30), which is 
a benefit-cost analysis. The American Petroleum Institute sponsored a study of the 
costs of meeting the EPA's proposed ozone standard in a multi-state area around 
the Great Lakes (26). Their estimate was as large as the EPA’s estimate for the 
entire country. No reference was made to the study, nor was any attempt made to 
reconcile the results. One would hope that future studies would attempt to acquire, 
analyze, and reconcile results of other major studies addressing relevant parts of 
future EPA benefit-cost analyses of air quality improvements. 

Peer Review 

Currently, peer review of the Section 812 studies is performed through unpaid 
review committees, mostly made up of academics, who meet and participate 
in conference calls periodically to consider issues that come up in the design. 


^One committee member estimated that costs of the implemented CA A Amendments could 
be 30% higher than shown in the report. 
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implementation, and writing of the review. Because of the necessarily limited 
attention that such reports can be given by unpaid committees, we recommend 
that their resources be supplemented by some paid reviewers with expertise in 
specific areas. This is an approach used by the EPA to do other reviews of agency 
work under the CAA, such as the reviews conducted of the voluminous Cri- 
teria Documents^ issued periodically to survey the literature relevant to setting 
new National Ambient Air Quality Standards, under Title I of the Act. 


REPRESENTING UNCERTAINTIES 
ABOUT COSTS AND BENEFITS 


I 




Over the years, benefit-cost analyses at the EPA have been getting more sophis- 
ticated in terms of their representation of the benefits of environmental improve- 
ments. Early analyses either ignored uncertainties, focusing on a best estimate 
for each of the parameters in the analysis, or identified rather arbitrary “low” and 
“high” estimates around a “best” estimate and used all the low or high estimates 
of parameters to create a “super low” and a “super high” benefit estimate, respec- 
tively. More recently, as exemplified in both the retrospective and the prospective 
studies, Monte Carlo simulation techniques were used. Such techniques apply a 
more logical and rigorous approach to propagating uncertainties in parameters 
through the various steps of the benefit analysis.*^ 

No one doubts that costs are uncertain. In fact, costs may be just as uncertain 
as benefits. However, costs are routinely considered at if they were certain. The 
same techniques used to analyze uncertainties in benefit estimates should be used 
to estimate uncertainties in costs. This will not be easy to do. For one thing, the 
databases containing the engineering data used to estimate costs of reducing air 
emissions typically offer only “best estimates,” with no information provided on 
the dispersion of such costs across plants. Second, because the cost estimates 
are engineering-derived rather than statistically based, they come from analyses 
that do not generate the kind of error distributions found in the statistically based 
epidemiological and valuation literature. 

These points lead to two research strategies. One would be to comb the eco- 
nomics literature for abatement cost functions estimated for various pollutants and 
sectors and use these instead of the engineering data. Another would be to comb 
the engineering literature for references to cost distributions across plants and use 
such information to develop cost ranges. 


*'^This approach involves first defining probability distributions for each parameter — often 
based on confidence intervals around estimated regression coefficients — e.g., from the effect 
of a change in pollution concentration on a health effect. Then simulations are run to draw 
parameter values from these distributions, computing benefits for each set of trial values 
drawn. The result is a benefit distribution that truly represents the underlying uncertainties 
in the parameters. 
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Up to now we have assumed that future Section 8 1 2 studies need to be benefit-cost 
studies. Yet, under certain reasonable conditions, cost-effectiveness analysis — 
dividing the cost of a particular regulation by a measure of the effectiveness of 
that regulation and then comparing various regulations (or regulatory options) 
according to this cost-per-unit effectiveness measure — may be almost as useful as 
benefit-cost analysis. 

The rationale for this strategy begins by noting that attaching monetary values 
to health effects is highly controversial, so avoiding doing so has an obvious 
attraction. Arguably, in the analysis of air pollution policies, in which such a large 
portion of the benefits are mortality-related and the morbidity benefits tend to 
move proportionally with changes in mortality, a physical measure of mortality 
benefits, such as “lives saved” or “life-years saved” may be a good proxy for all 
health effects. In this case, a cost-per-Ufe-saved measure may be a useful basis for 
discriminating among various policy initiatives. 

There are drawbacks to this approach, however. First, and most importantly, 
only with benefit-cost analysis can any normative claim be made, i.e., whether 
the contemplated or already implemented policy makes economic sense. When 
a benefit-cost analysis of a contemplated action yields negative net benefits, one 
can make the statement that, on efficiency grounds (and subject to many caveats), 
society would be made worse off by advancing that particular option. If option 
A has greater (positive) net benefits than option B, then one can say that both 
options increase social welfare, but that A is the better option, again, solely from 
the criterion of efficiency. With only a cost-effectiveness analysis, in contrast, if 
one finds that option A is more cost-effective than option B (cost per life saved is 
lower for A than B), one can only conclude that A is preferred to B, not that A is 
socially beneficial on net. 

Second, for a cost-effectiveness analysis to be a reasonable substitute for benefit- 
cost analysis, one category of benefits (e.g., the mortality benefits) needs to be a 
large fraction of all benefits. In the context of the CAA, we have argued above that 
there is some reason to expect that the VSL estimates will be coming down. This 
means that morbidity benefits may assume a greater share of the total and that, 
as a consequence, the cost-effectiveness strategy may be less attractive. Further, 
to the extent that ecological benefits are involved, human mortality benefits may 
be a poor proxy for them. Certainly, once these as yet poorly understood benefit 
areas are better understood, the cost-effectiveness strategy will be less compelling 
as applied to the CAA. 


CONDUCT VALUE OF INFORMATION ANALYSIS 


Finally, many benefit-cost analyses, including the Section 812 studies, are also 
useful in helping to set a research agenda. Once one goes to the trouble and ex- 
pense of conducting such a study, it is not much more complicated to estimate 
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whether a reduction in the uncertainty of various parameters would materially af- 
fect the ultimate conclusions of the analysis. In other words, one can determine 
which parameters contribute the most to benefit (or cost) uncertainty and would, 
therefore, be the most important for reducing uncertainty. Implicitly at least, the 
EPA conducted such a thought process in developing a list of key areas of future 
research to improve future Section 812 studies. It might be appropriate to for- 
malize that process to get a deeper understanding of the value of proposed new 
research. 


CONCLUDING THOUGHTS 


I 


The EPA’s Section 812 studies are unique in government. To our knowledge, no 
other agency provides broad-scope benefit-cost analyses for statutes it implements 
that are as carefully developed or reviewed. Whereas benefit-cost analyses are now 
required for all major regulations, the Section 812 analysis serves to integrate such 
analyses while taking a national and forward-looking perspective on air pollution 
control. 

Subject to some methodological qualifications, the two Section 812 studies al- 
ready completed indicate that the aggregate benefits of past and ongoing policies 
to improve air quality have clearly exceeded the costs incurred by industry, tax- 
payers, and consumers. Whether benefits exceed costs by a factor of 40 or 4 or 
less, the professional consensus is that, overall, the nation received good value for 
the resources it invested in improving air quality over the past three decades. Yet 
we don’t know if that v-alue could have been far higher by changing or eliminat- 
ing certain inefficient elements of the CAA. One of the challenges of a second 
prospective study is to answer that question. 

Another challenge in conducting a new Section 812 study, which is less con- 
strained by Congressional mandate than the first two, is to develop the analysis in 
such a way as to have the maximum impact on the design and implementation of 
future air pollution control policies. Specifically, wc have recommended the goal 
of increasing the net benefits of future air quality regulations. We believe the best 
way to accomplish this is to focus the study on already mandated but not fully im- 
plemented polices where significant future costs are anticipated and, particularly, 
to focus on potential new policies not yet mandated by law or regulation. We have 
recommended specific candidates for study, but clearly a fuller debate on the issue 
is appropriate. 

We also believe the EPA faces a number of key challenges in incorporating the 
most recent research in a new Section 812 study and in making the Section 812 
study effort a model of how good benefit-cost analyses are designed, performed, 
and reviewed. The EPA should resist the temptation to sit on its laurels with respect 
to the positive outcome of the Section 812 studies, recognize the possibility that 
the CAA can do a better job of delivering benefits to residents of the United States, 
and endeavor to keep the analysis at ffie forefront of the rapidly changing social 
and physical sciences it must draw upon. 
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Finally, the EPA should take the opportunity afforded by the new Section 812 
study to do something it has rarely done before: actually use benefit-cost analysis 
prospectively as a planning tool to establish priorities for its regulatory and legisla- 
tive agenda with respect to air pollution issues. Such an action would demonstrate 
to Congress that the EPA is serious about delivering efficient regulations and other 
policy initiatives to the American public. 


Visit the Annual Reviews home page at www.annualreviews.org 
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Synopsis of the Particle Epidemiology Reanalysis Project 


BACKGROUND 

Epidemiologic work conducted over several 
decades has suggested that long-term residence in 
cities with elevated ambient levels of air pollution 
from combustion sources is associated with 
increased mortality. Subsequently, two prospec- 
tive cohort studies, the Six Cities Study (as 
reported in Dockery et al 1993) and the American 
Cancer Society (ACS) Study (as reported in Pope et 
al 1995) estimated that annual average all-cause 
mortality increased in association with an Increase 
in fine particles (all particles less than 2.5 pm in 
median aerodynamic diameter IPM2.5I). 

As part of the Six Cities Study, Dockery and col- 
leagues (1993) had prospectively followed a cohort 
of 8,111 adult subjects in northeast and midwest 
United States for 14 to 16 years beginning in the 
mid-1970s. The authors found that higher ambient 
levels of Tme particles and sulfate (SO^^”) wore 
associated with a 26% increase in mortality from 
all causes when comparing the most polluted to the 
least polluted city, and that an increase in fine par- 
ticles was also associated with increased mortality 
from cardiopulmonary disease. The relative risks 
in all-cause mortality were associated with a differ- 
ence (or range) in ambient fine particle concentra- 
tions of 18.6 pg/m^ and a difference of ambient 
sulfata conconlrations of 8.0 pg/m^. comparing the 
least polluted city to the most polluted city. 

In the much larger ACS Study, Pope and col- 
leagues (1995) followed 352,130 adult subjects in 
154 US cities beginning in 1982 and ending in 1989 
[3 cities did not overlap between the 151 and 
50 cities studied, resulting in a total of 154 cities). 
Again, higher ambient levels of fine particles were 
associated with increased mortality from al) causes 
and from cardiopulmonary disease in the SO cities 
for which fine particle data were available (sam- 
pled from 1979 to 1983). Higher ambient sulfate 
levels were associated with increased mortality 


from all causes, cardiopulmonary disease, and 
lung cancer in the 151 cities for which sulfate data 
were available (sampled from 1980 to 1982). The 
diQercoc% between all-cause mortality in the most- 
polluted city and the least-polluted city was 17% 
and 15% for fine particles and sulfate, respectively 
(with a range of 24.5 pg/m^ for fine particles and of 
19.9 i^m‘^ for sulfate). 

Both of these studies came under intense scru- 
tiny in 1997 when the EPA used the results tu sup- 
port new National Ambient Air Quality Standards 
for fine particles and to maintain the standards for 
particles less than 10 pm in median aerodynamic 
diameter (PM^q) already in effect. Members of 
Congress and industry, the scientific community 
and others interested in regulation of air quality 
scrutinized the studies' methods and their resolts. 
Some insisted that any data generated using fed- 
eral funding should be made pnbiic. Others 
argued that these data had been gathered with 
assurances of confidentiality for the individuals 
who had agreed to participate and that the concept 
of public access to federally funded data did not 
take into acconnt the intellectual property rights of 
the investigators and their supporting institutions. 
To address the public controversy, Harvard Uni- 
versity and the ACS requested that the Health 
Effects institute organize an independent rcanal- 
ysis of the data from these studios, Both institu- 
tions agreed to provide access to their data to a 
team of analysts to be selected by HE! through a 
competitive process. 


APPROACH 

To conduct the reanalysis, the HEI Board of 
Directors, with support from the EPA, industry, 
Congress, and other stakeholders, appointed an 
Expert Panel chaired by Dr Arthur Upton from the 
University of Medicine and Dentistry of New 
Jersey and former Director of the National Cancer 


This Statement, prepared by the Health Effects Insiimie, tea summaty of a research project conducted by the Reanalysis Tfeara, led by Dr 
Daniel ICicwski at the University of Ottawa. TbeldJowing Special Rep^conudns the detailed Investigators' Report (Summary.Inlioduc- 
licn.andPartslandl!), Commentary on the project prepared by a special panel utihe Institute's Health Review Committee, and Comments 
on the Heanalysis Project by the Original Investigalois (Dis Douglas W Dockery C Arden HI cl al). 
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Institute. The Expert Panel selected competitively 
a Reanalysis Team — led by Dr Daniel Krewski of 
the University of Ottawa — and oversaw all aspects 
of the team's work. They were assisted in their 
oversight efforts by a broad-based Advisory Board 
of knowledgeable stakeholders and scientists who, 
in the project’s early stages, provided extensive 
advice to the Expert Panel on the key questions to 
be analyzed. The final results of the Reanalysis 
Team were intensively and independently peer 
reviewed by a Special Panel of the HEI Health 
Review Conimittee, which was chaired by Dr Mil- 
licent Higgins of the University of Michigan. 

The overall objective of w’hat became the Par- 
ticle Epidemiology Reanalysis Project was to con- 
duct a rigorous and independent assessment of 
the findings of the Six Cities and ACS Studies of 
air pollution and mortality. This objective was 
met in two parts. In Part /: RepiicaUon and Valida- 
tion, the Reanalysis Team sought to replicate the 
original studies via a quality assurance audit of a 
sample of the original data and to validate the 
origioal numeric results. In Part 11: Sensitivity 
Analyses, they tested the robustness of the orig- 
inal analyses to alternate risk models and analytic 
approaches. 


RESULTS AND IMPLICATIONS 


PART I: REPLICATION AND VAI.IDATION 

• An extensive audit of the study popniation 
data for both the Six Chios and ACS Studios 
and of tho air quality data in the Six Cities 
Study revealed the data to be of generally high 
quality with a few exceptions. In both studies, 
a few errors were found in the coding and 
inclusion of certain subjects; when those sub- 
jects were included in the analyses, they did 
not materially change the results as originally 
reported, Because the air quality data nsed in 
the ACS Study could not be audited, a sepa- 
rate air quality database was constructed for 
the sensitivity analyses described in Part 11. 

• The Reanalysis Team was able to replicate the 
original results in both studies using tho same 
data and statistical methods as used by the Orig- 
inal Investigators. The Reanalysis Tram con- 
firmed the original point estimates: For the Six 


Cities Study, they reported the relative risk of 
mortality from all causes associated with an 
increase in fine particles of 18.6 pg/m^ as 1.28. 
close to the 1.26 reported by the Original Inves- 
tigators. For the ACS Study, the relative risk of 
mortality from all canses associated with an 
increase in fine particles of 24.5 pg/m^ was 1.18 
in the reanalysis, close to the 1.17 reported by 
the Original Investigators. 

PART II: SENSITIVITY ANALYSES 

Once tho original results of the studies had been 
validated, the Reanalysis Team sought to test an 
array of different models and variables to deter- 
mine whether the original results would remain 
robust to different analytic assumptions. 

• First, the Reanalysis Team used the standard 
Cox model used by the Original Investigators 
and included variables in the model for which 
data were available from both original studies 
but had not been used in the published analy- 
ses (eg. physical activity, lung function, mari- 
tal status}. The Reanalysis Team also designed 
models to include interactions between vari- 
ables. None of these alternative models pro- 
duced results that materially altered the 
original findings. 

• Next, for both the Six Cities and ACS Studies, 
the Reanalysis Team sought to test the possi- 
ble effects of fine particles and sulfate on a 
range of potentially susceptible subgroups of 
the population. Althotigh different subgroups 
did show some variation in their estimated 
effects, the results were not statistically signif- 
icant with one oxception. Tlic estimated 
effects of fine particles did appear to vary with 
educational level: the association between an 
increase in fine particles and mortality tended 
to be higher for individuals without a high 
school education than for those who had com- 
pleted high school or for those with more than 
a high school oducatioii. 

• In the ACS study, the Reanalysis Team tested 
whether tho relationship between ambient 
concentrations and mortality was linear. They 
found some indications of both linear and 
nonlinear relationships, depending upon the 
analytic technique used, suggesting that the 
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issue of concentration-response relationships 
deserves additional analysis. 

• In the Six Cities Stndy where data were avail- 
able, the Reanalysis Team tested whether 
effect estimates changed when certaia key risk 
factors (smoking, body mass indinc, and air 
pollution) were allowed to vary over time. 
One of the criticisms of both original studies 
has been that neither analyzed the effects of 
change in pollutant levels over time. In gen- 
eral. the reanalysis results did not change 
when smoking and body mass index were 
allowed to vary over time. The Reanalysis 
Team did find for the Six Cities Study, how- 
ever, that when the general decline in fine par- 
ticle levels over the monitoring period was 
included as a time-dependent variable, the 
association between fine particles and all- 
cause mortality dropped substantially, but the 
effect continued to be positive and ^listi- 
cally significant 

• Using its own air quality dataset constructed 
from historical data to test the validity of the 
original ACS air quality data, the Reanalysis 
Team found essentially the same results. 

• Any future analyses using the sulfate data 
should take into account the impact of artifac- 
tual sulfate. Sulfate levels with and without 
adjustment differed by about t0% for the Six 
Cities Study. Both the CTriginal ACS Study air 
quality data and the newly constructed 
dataset contained sulfate levels inflated hy 
approximately 30% due to artifactual sulfate. 
For the Six Cities Study, the relative risks of 
mortality were essentially unchanged with 
adjusted or unadjusted sulfate. For the ACS 
Study, adju.sting for artifactual sulfate resulted 
in slightly higher relative risks of mortality 
front all causes and cardiopulmonary disease 
compared with unadjusted data. The relative 
risk of mortality from lung cancer was lower 
after the data had been adjusted. 

• Because of the limited statistical power to con- 
duct most sensitivity analyses for the Six Cit- 
ies Study, the Rcanalysis Team conducted the 
majority of its sensitivity analyses using only 
the ACS Study dataset with 154 cities. In that 
dataset, when a range of city-level (ocologic) 
variables (eg, population change, measures of 
income, maximum temperature, number of 


hospital beds, water hardness) were included 
in the analyses, the results generally did not 
change. Two exceptions were that associations 
for both fine particles and sulfate were 
reduced when city-level measures of popula- 
tion change or sulfur dioxide were included in 
the model. 

• A major contribution of the Rcanalysis Project 
is the recognition that both pollulant variables 
and mortality appear to be spatially correlated 
in the ACS Study dataset. If not identified and 
modeled correctly, spatial correlation could 
cause snbstantial errors in both the regression 
coefficients and their standard errors. The 
Rcanalysis Team identified several methods 
for dealing with this, all of which resulted in 
some reduction in the estimated regression 
coefficients. The full implications and inter- 
pretations of spatial correlations in these anal- 
yses have not been resolved and appear to be 
an important subject for future research. 

• When the Reanalysis Team sought to take into 
account both the underlying variation from 
city to city (random effects) and the spatial 
correlation between cities, only sulfur dioxide 
as Q city-level variable continued to decrease 
the originally reported associations between 
mortality and fine particles or sulfate. This 
effect was more pronounced for sulfate. 

• When the Rcanalysis Team conducted spatial 
analyses of sulfur dioxide, the association 
between sulfur dioxide and mortality per- 
sisted after adjusting for sulfate, fine particles, 
and other variables. 

• Asa result of these extensive analyses, the 
Rcanalysis Team was able to explain much of 
the variation between cities, but some unex- 
plained city-lo-city variation remained. 


CONCLUSIONS 

The Reanalysis Team designed and imple- 
mented an extensive aud sophisticated series of 
analyses that included a set of new variables, all 
the gaseous copoHutants, and the first attempts to 
apply spatial analytic methods to test the validity 
of the data and the results from the Six Cities 
Study and the ACS Study. Overall, the reanalyses 
assured the quality of the original data, replicated 
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the original results, and tested those results against 
alternative risk models and analytic approaches 
without substantively altering the original find- 
ings of an association between indicators of partic- 
ulate matter air pollution and mortality. 

At the same time, the reanalyses did extend and 
challenge orir understanding of the original results 
in several important ways. 

• The Roannlysis Team identified a possible 
modifying effect of education on the relation 
between air quality and mortality in that esti- 
mated mortality effects increased in the sub- 
group with less than high school education. 

• The use of spatial analytic methods suggested 
that, when the analyses controlled for correla- 
tions among cities located near one anothca-, the 
associations between mortality and fine parti- 
cles or sulfate remained but were diminished. 

• An association between sulfur dioxide and 
mortality was observed and persisted when 
other possible confounding variables were 
included; furthermore, when sulfur dioxide 
was included in models with fine particles or 
sulfate, the associations between these pollut- 
ants ffiuc particles and sulfate) and mortality 
diminished. 

In reviewing these results, the Special Panel of 
the HEI Health Review Committee identified the 
following factors to consider when interpreting 
the results from the Roanalysis Team. 

• The inherent limitations of using only six cit- 
ies. understood by tho Original Investigators, 
should be taken into account when interpret- 
ing results of iho Six Cities Study. 

• The Reanalysis Team did not use data 
adjusted for artifactual snifate for most alter- 
native analyses. When they did use adjusted 


sulfate data, relative risks of mortality from 
ail causes and cardiopulmonary disease 
increased. This result suggests that more 
aimlyses with adjusted sulfate might result in 
somewhat higher relative risks associated 
ivith sulfote. 

• Findings from spatial analyses applied to the 
ACS Study data need to be interpreted with 
caution; the spatial adjustment may have 
overadjusted the estimated effect for regional 
pollutants such as fine particles and sulfate 
compared with the effect estimates for more 
local pollutants such as sulfur dioxide. 

• After the Reanalysis Team completed its spa- 
tial analyses, residual spatial variation was 
still noticeable; this finding suggests that 
additional studies might further refine our 
understanding of the spatial patterns in both 
air pollution and mortality. 

• No single epidemiologic study can bo the 
basis for determining a causal relation 
between air pollution and mortality. 

In conclusion, the Reanalysis Team interpreted 
ihoir findings to suggest that increased relative 
risk of “mortality may be allxibuted to more than 
one component of the complex mix of ambient air 
pollutants in urban areas in the United States”. 
The Review Panel concurs, tn the alternative anal- 
yses of the ACS Study cohort data, the Reanalysis 
Team identified relatively robust associations of 
mortality with fine particles, sulfate, and sulfur 
dioxide, and they tested these associations in 
nearly every possible manner within the Umita- 
tion.s of the datasets. Future Investigations of 
these issues will enhance our understanding of 
the effect of combustion-source air pollutants (eg. 
fine particles, sulfate, and sulfur dioxide] on 
public health. 
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The Cost-Effectiveness of Environmental Approaches 
to Disease Prevention 

Dave A. Chokshi, M.D., and Thomas A. Farley, M.D., M.P.H. 


H ow can society prevent the 
most disease and deaths 
per dollar spent? This question 
arose throughout the debate on 
U.S. health care reform and will 
continue to drive decision mak- 
ing as health care funding be- 
comes increasingly constrained. 
In an atmosphere of austerity, 
demonstrating the cost-effective- 
ness of preventive health inter- 
ventions becomes particularly im- 
portant. 

Although preventive approach- 
es to disease are intuitively ap- 
pealing — and frequently pre- 
sented as a way to reduce costs 
— analyses have suggested that, 
as a whole, they’re no more cost- 
effective chan therapeutic interven- 
tions.^ But are some preventive 
approaches more cost-effective 
than others? The National Com- 
mission on Prevention Priorities 
attempted to address this ques- 
tion, ranking clinical preventive 
services in terms of cost-effec- 
tiveness and “clinically prevent- 
able burden” of disease.^ Yet some 
preventive services, such as to- 
bacco taxes or water fluoridation, 
are not delivered in health care 
settings. Understanding whether 
certain approaches are more cosr- 
effeccive than others requires a 
framework for categorizing pre- 
ventive interventions. 

Medicine traditionally classifies 


preventive interventions on the ba- 
sis of disease course: primary pre- 
vention aims to prevent new cases 
of disease; secondary prevention 
and tertiary prevention mitigate 
the effects of existing disease. 
We propose two overlapping di- 
mensions to further characterize 
primary preventive interventions: 
environmental versus person- 
directed, indicating whether the 
proximate target is an element of 
the environment or an individu- 
al, and clinical versus nondinical, 
indicating where an intervention 
takes place. Separating person- 
directed from environmental intet- 
ventions permits the comparison 
of prevention conducted individ- 
ual by individual (e.g., cancer 
screening) with prevention that 
acts on persons indirectly by al- 
tering the physical or social envi- 
ronment (e.g., a ban on trans fats). 
Whether an intervention takes 
place within a health care setting 
or elsewhere has implications for 
resource allocation, since fund- 
ing streams for clinical and non- 
clinical interventions tend to be 
distinct. Some nondinical inter- 
ventions, such as syringe-exchange 
programs, are person-directed, but 
all environmental interventions are 
nondinical. 

Because reaching individuals 
directly is generally more expen- 
sive than changing an environ- 


mental element, we hypothesized 
that unless a person-directed In- 
tervention was very effective {like 
childhood immunization, for ex- 
ample), environmental interven- 
tions would generally be more 
cost-effective. We further hypoth- 
esized that it mattered where an 
intervention was delivered and 
that nondinical, person-directed 
interventions would be more cost- 
effective than clinical interven- 
tions. To test these hypotheses, 
we conducted a comparative analy- 
sis of the cost-effectiveness of 
environmental, nondinical but 
person-directed, and clinical pre- 
ventive interventions. 

We analyzed the contents of 
the Tufts Medical Center Cost- 
Effectiveness Analysis (CEA) Reg- 
istry, which contains information 
on 2815 cost-effectiveness analy- 
ses published through December 
2011. Costs per quality-adjusted 
life-year (QALY, a unit of measure 
for survival that accounts for the 
effects of suboptimal health sta- 
tus) are reported after conversion 
to 2011 U.S. dollars. Only cost- 
utility analyses — which permit 
comparison of programs ad- 
dressing different health prob- 
lems by converting health out- 
comes into a common metric 
— are included in the registry. 
We excluded studies that didn’t 
report on an intervention meet- 
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Cost-Effectiveness of Categories of Preventive Interventions. 


ing the definition of primary 
prevention and categorized the 
remaining studies as environ- 
mental, clinical, or nonclinical 
but person-directed (for complete 
methods, see the Supplementary 
Appendix, available with the full 
text of this article at NEJM.org). 

According to our definitions, 
the CEA Registry contained 401 
studies of clinical prevention, with 
1259 associated cosc-e^ectiveness 
ratios; 37 studies of nonclinical, 
petson-direcced prevention, with 
83 associated cost-effectiveness 
ratios; and 31 studies of environ- 
mental prevention, with 59 asso- 
ciated cost-effectiveness ratios. 
Environmental interventions were 
generally more cost-effective than 
clinical interventions or nonclini- 
cal, person-directed interventions 
(see graph); the proportion that 
were cost-saving was higher 
among environmental interven- 
tions (46%) than among clinical 
interventions (16%, P<0.001) or 
nonclinical, person-directed inter- 
ventions (13%, P<0.001). The dis- 
tribution of cost-effectiveness ra- 


tios was similar for clinical 
interventions and nonclinical, per- 
son-directed interventions. TWenty- 
seven of the 59 cost-effectiveness 
ratios for environmental inter- 
ventions (46%) indicated that the 
interventions were cost-saving; an 
additional 10 environmental in- 
terventions (17%) cost less than 
$10,000 per QALY, and 15 (25%) 
cost $10,000 to $50,000 per QALY. 
As a point of reference, $50,000 
to $100,000 per QALY is often 
used as the upper limit for favor- 
able cost-effectiveness ratios. 

In an environmental model of 
prevention, people’s behavior is 
influenced by their physical and 
social environment.^ It can be far 
less expensive to alter an envi- 
ronmental element to which many 
people are exposed than to inter- 
act with each person directly. 
Even if the effect of an altered 
environment on each person is 
small, the cumulative population 
effect can be large; cost-effective- 
ness can be favorable because the 
cost per person reached is small. 
Eor instance, Smith-Spangler et al. 


estimated that, as compared with 
the status quo, a tax on sodium 
that reduces population sodium 
intake by 6% would reduce heart 
disease and stroke incidence, in- 
crease QALYs by 1.3 million, and 
save $22.4 billion over the life- 
lime of adults who are currently 
40 to 85 years of age.* 

We were surprised to find that 
nonclinical, person-directed pre- 
ventive interventions were not 
more cosi-effecrlve than clinical 
interventions. Although the ab- 
sence of a discerned effect may 
not indicate a true absence of 
effect, this finding suggests 
rhat the “environmental” char- 
acter of an intervention may be 
more important than the “non- 
clinical” character in determin- 
ing cost-effectiveness. Environ- 
mental change may have initial 
costs followed by lasting effect 
(e.g., building recreational facili- 
ties to promote physical activity), 
whereas person-directed interven- 
tions have continued costs (e.g., 
exercise programs). Furthermore, 
many environmental interven- 
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lions are low-cost because they’re 
implemented by regulation (e.g., 
smoke-free-air laws) or are exe- 
cuted centrally (e.g., food forti- 
fication with folic acid). Some 
environmental interventions, such 
as excise taxes, may generate gov- 
ernment revenue that can offset 
costs or be used for health pro- 
grams. 

Our finding that the environ- 
mental interventions studied were 
the most likely to be cost-saving 
doesn’t necessarily mean that all 
environmental interventions are 
cost-effective. Some may be ex- 
pensive to implement and benefit 
few people — for example, 
building-safety regulations that 
prevent extremely rare injuries. 
The finding suggests, however, 
that there may be more cost-effec- 
tive environmental interventions 
than are currently recognized and 
that such interventions deserve 
more attention. 

Our analysis could be biased 
by underreporting of studies of 
ineffective environmental interven- 
tions, leading to an overestimate 
of favorable cost-effectiveness. On 
the other hand, published stud- 
ies are more likely to investigate 
contentious topics, rather than 
interventions widely known to be 
cost-effective. Also, wq found 
significantly fewer studies on en- 
vironmental and nonclinical, per- 
son-directed interventions than 
on clinical interventions. More 
generally, cost-effectiveness is 
predicated on an initial demon- 
stration of effectiveness, which is 


often difficult and analytically 
fraught for preventive interven- 
tions, particularly environmental 
ones. Assessing the value of pre- 
vention is more difficult than 
evaluating treatments for estab- 
lished disease, because the long 
time horizon for clinical end 
points introduces considerable 
uncertainty about benefits. 

Our findings have important 
implications for resource alloca- 
tion. Environmental prevention is 
key to addressing the growing 
disease burden and cost of chron- 
ic illnesses. For example, in New 
York City, an environmental ap- 
proach to chronic-disease preven- 
tion included increased tobacco 
taxes, a comprehensive smoke- 
free-air law, mass-media cam- 
paigns against smoking and sugar- 
sweetened beverages, the banning 
of trans fats from restaurants, 
and a restaurant calorie-labeling 
initiative. It has been estimated 
that the antismoking initiatives 
alone reduced the number of 
smokers in the city by 450,000 
over a decade and the number of 
smoking-related deaths by 1500 
per year. But increased investment 
in environmental interventions 
should not be pitted against per- 
son-directed interventions; in most 
cases, the two work synergisci- 
cally, as they did in effecting 
large decreases in mortality from 
cardiovascular disease in the sec- 
ond half of the 20th century.® 

The paucity of studies on the 
cost-effectiveness of environmen- 
tal preventive interventions im- 


pedes their broader adoption. 
Unlike other forms of economic 
evaluation, cost-effeaiveness stud- 
ies can demonstrate value through 
direct comparison of alternative 
interventions. The scientific lit- 
erature now points to the value 
of implementing preventive envi- 
ronmental interventions that are 
cost-saving and conducting addi- 
tional cost-effectiveness studies 
of such interventions. 

The views expressed in this article ace 
those of the authors and do not necessarily 
represent those of the New York City De- 
partment of Health and Mental Hygiene. 

Disclosure forms provided by the au- 
thors are available with the full text of this 
article at NEJM.org. 
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Abstract 

Background — In recent years (2000 to 2007), ambient levels of fine particulate matter (PM 2 , 5 ) 
have continued to decline as a result of interventions, but the decline has been at a slower rate than 
previous years (1980 to 2000). Whether these more recent and slower declines of PM 2.5 levels 
continue to improve life expectancy and whether they benefit all populations equally is unknown. 

Methods — We assembled a dataset for 545 U.S. counties consisting of yearly county-specific 
average PMi.s, yearly county-specific life expectancy, and several potentially confounding 
variables measuring socioeconomic status, smoking prevalence and demographic characteristics 
for the years 2000 and 2007. We used regression models to estimate the association between 
reductions in PM2.5 and changes in life expectancy for the period 2000 to 2007. 

Results — A decrease of 10 pg/m^ in the concentration of PM 2.5 was associated with an increase 
in mean life expectancy of 0.35 years SI>= 0.16 years, p = 0.033). This association was stronger in 
more urban and densely populated counties. 
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Conclusions — Reductions in PM 2.5 were associated with improvements in life expectancy for 
the period 2000 to 2007. Air pollution control in the last decade has continued to have a positive 
impact on public health. 

Since the 1970s, enactment of increasingly stringent air quality controls has led to 
improvements in ambient air quality in the United States at costs that the U.S. 

Environmental Protection Agency (EPA) has estimated as high as S25 billion per year.* 
However, even with the well-established link between long-tenn exposure to air pollution 
and adverse effects on health,^ the extent to which more recent regulatory actions have 
benefited public health remains in question. 

Air pollutant concentrations have been generally decreasing in the U.S., with substantial 
differences in reductions across metropolitan areas. Levels of fine particulate matter air 
pollution (particulate matter < 2.5 pm in aerodynamic diameter, PM 2 . 5 ) remain relatively 
high in some areas. In a 2010 study, the EPA estimated that 62 U.S. counties, accounting for 
26% of their total study population, had PM 2.5 concentrations not in compliance with the 
National Ambient Air Quality Standards (NAAQS).-* 

Reductions in particulate matter air pollution are associated with reductions in both 
cardiopulmonary and overall mortality.- In the mid-1990s, the Harvard Six Cities Study‘s 
and the American Cancer Society (ACS) study^ reported associations of cardiopulmonary 
monality risk with chronic exposure to fine particulate air pollution while controlling for 
smoking and other individual risk factors. Reanalysis and extended analyses of these studies 
have confirmed that fine particulate air pollution is an important independent environmental 
risk factor for cardiopulmonary disease and mortality.^*^ Additional cohort studies, 
population-based studies, and short-term time-series studies have also shown associations 
between reductions in air pollution and reductions in human mortality.*^ '^* More recently, 
studies have suggested an association between PM 2.5 and life expectancy, a well- 
documented and important measure of overall public health.-**"^^ 

As our primary analysis, we estimate the association beriveen changes in PM 2,5 and in life 
expectancy in 545 U.S. counties during the period 2000 to 2007. This period is of particular 
interest, as the EPA restarted wide collection of PM 2.5 data in 1999-2000, after stopping the 
nationwide PM 2.5 monitoring program during the mid-1980s and most of the 1990s. . In 
secondary analy.ses, we extended to 2007 the data and statistical analysis originally reported 
by Pope and colleagues’^ for the period 1980-2000, and investigated whetlier the 
relationship reported by Pope et al^^ persists in the more recent years. 

METHODS 

Data 

We constructed and analyzed three data sets to estimate the association between changes in 
life expectancy and changes in PM2.5 during the period 2000 to 2007 in 545 counties 
(Dataset 1), and to investigate whether the association previously reported by Pope et al^-^ 
persists when the data on the same 211 counties are extended to the year 2007 (Datasets 2 
and 3). 

Dataset 1 included information on 545 U.S. counties for the years 2000 and 2007. These 
counties include all counties with available matching PM 2,5 data for 2000 and 2007. 
Additionally, unlike previous work in W'hich counties were located only in metropolitan 
areas,^^ Dataset I is comprised of counties in both metropolitan and non-metropolitan areas. 
Figure 1 shows the counties in this dataset shaded according to life expectancy in 2000 and 
2007. Variables in this dataset were available at the county level, for both 2000 and 2007, 
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and included; life expectancy, PM 2 . 5 , per capita income, population, propoitions who were 
high school graduates, and proportions who were white, black, or Hispanic. Because data on 
smoking prevalence were not available for all 545 counties, we used age-standardized death 
rates for lung cancer and chronic obstructive pulmonary disease (COPD) as proxy variables 
for smoking prevalence Death rates were calculated in 5-year age groups and age- 
standardized for the 2000 U.S. population of adults 45 years of age or older. Daily PM 2.5 
data were obtained from the EPA’s Air Quality System (AQS - http://www.epa.gov/ttn/ajrs/ 
airsaqs/detaildata/dowTiloadaqsdata.htm). Daily PM 2.5 levels for each county were averaged 
across monitors within that county using a trimmed mean approach; those daily county-level 
means were further averaged across days to obtain a county-specific yearly PM 2.5 average.^^ 

County-level life expectancies were calculated by applying a mixed-effects spatial Poisson 
model to mortality data from the National Center for Health Statistics (NCHS) and 
population data from the U.S. Census to obtain robust estimates of the number of deaths in 
each county.'^® These estimated counts were then used to calculate county life expectancies 
using standard life table techniques, which we discuss in more detail in the eAppendix 
(Section A). 

Socioeconomic and demographic variables were obtained from the U.S. Census and the 
American Community Survey e.xcept per capita income, which was obtained from the 
Bureau of Economic Analysis. All yearly income variables were adjusted for inflation with 
2000 as the base year. Age-standardized death rates for lung cancer and COPD were 
calculated using mortality data from NCHS using death rates for 2005 to serve as a proxy 
for 2007 (NCHS data for 2007 was not readily available). Lastly, data on smoking 
prevalence (proportion of the population who are current smokers) w’ere available from the 
Behavioral Risk Factor Surveillance System in both 2000 and 2007 for 383 of the 545 
counties. 

Dataset 2 included data for the year 1980 and the year 2000 for the same 211 U.S. counties 
included in the 5 1 metropolitan statistical areas (MSAs) previously analyzed by Pope and 
colleagues.^^ This dataset is identical to that iu the paper by Pope et al,^^ where it is 
described in more detail. 

Dataset 3 extended Dataset 2 to 2007. All data were available at the county level except for 
PM 2 . 5 , w'hich for the year 1980 was available only at the MSA level and for the year 2007 
was available at the county level for only 1 1 3 of the 2 1 1 counties originally included in 
Pope et al,^^ Thus, for the year 2007, we assigned the same PM 2,5 values to all the counties 
that shared an MSA, consistent with the previous analysis.^^ Details and results pertaining to 
Datasets 2 and 3 are summarized in the eAppendix (Section Bl). 

Statistical Analysis 

Cross-sectional and first-difference linear regres-sion models were fitted to all three datasets. 
Specifically, we regressed life expectancy versus PM 2.5 levels across counties separately for 
the years 1980 (Dataset 2), 2000 (Datasets I and 2), and 2007 (Datasets 1 and 3). We then 
regressed changes in life expectancy over the years 2000 to 2007 (Datasets 1 and 3), 1980 to 
2000 (Dataset 2), and 1980 to 2007 (Dataset 3) versus changes in PM 2,5 over those same 
periods adjusted for changes in the socioeconomic, demographic, and proxy smoking 
variables outlined above. Additionally for our largest dataset (Dataset 1: 545 counties, 2000 
to 2007), we also performed several stratified and weighted analyses. More specifically, we 
estimated the effect of changes in PM 2.5 on life expectancy in models stratified by: 1 ) 
percentage of the population with an urban residence in 2000 ; 2 ) population density in 2000 ; 
3), land area in 2000; 4) PM 2.5 levels in 2000; 5) 5-year in-migration in 2000; and 6 ) change 
in average yearly temperature over the entire period. These stratified analyses allowed us to 
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examine whether PM2.5 effects on life expectancy were different in counties with particular 
demographic or weather characteristics. The sensitivity of our results to model specification 
was further assessed by fitting models weighted by: I ) total population; 2) year 2000 
population density; and 3) inverse land area. We included direct measures of the change in 
prevalence of smoking for the subgroup of counties with matching data on smoking 
prevalence (383 out of 545), and fit separate models for men and women to determine if 
effects differed by sex. To account for the correlation due to clustering of counties in the 
same MSA, robust clustered standard errors were calculated for all models.-^'^' Specifically, 
the variance of the vector of estimated regression coefficients, Pesc^ ’s given by: Var(pest) == 
(XTxr'CXTVeaX) (X^X)-’, where Vest is a block-diagonal matrix with non-zero blocks 
Vqj ~ (yj “ ~ w'here j indexes the MSAs. Pest is equal to the ordinary least 

squares estimator. Models were estimated using either REGRESS in STATA version 1 1 .0, 
!m() in R version 2.1 1.1, orPROC SURVEYREG in SAS version 9.2. 


RESULTS 


We report the results of our primary analysis, which estimated the cross-sectional 
relationship between life e.xpectancy and PM2 5, and betw^een changes in life expectancy and 
changes in PM2.5, for the period 2000 to 2007 in 545 US counties (Dataset 1). Results of the 
secondary analyses of the counties studied by Pope cl aP-’ using Datasets 2 and 3 are in the 
eAppendix (Section B; eTables !a,b and 2a,b). Table 1 lists the summary statistics for the 
variables in Dataset 1 . In 2000, 1 89 of the 545 counties had a PM2.5 level greater than the 
current 3-year NAAQS level of 1 5pg/m^; by 2007 only 48 of those 1 89 were not in 
compliance with the NAAQS. On average, P-M2.5 levels decreased at a rate of 0.22 pg/m^ 
per year, a rate 33% lower than observed in the 21 1 counties analyzed for the period 1980 to 
2000 (0.33 pg/ni^ peryear).--^ 

Figures 2A and 2B show life expectancies plotted against PM2,5 levels for the years 2000 
and 2007. Consistent with Pope et al,^^ cross-sectional regression models showed a negative 
association between life expectancy and PM2.5 in both years. Details are summarized in the 
eAppendix (Section C). 

Figures 2C and 2D show changes in life expectancy plotted against changes in PM2.5 levels 
for 2000 to 2007. We also plotted the estimated regression lines under Models I and 3 of 
Table 2, defined below. 

Table 2 summarizes estimated regression coefficients for the association between changes in 
PM2.5 and changes in life expectancy for 545 counties for 2000 to 2007 for selected 
regression models. When controlling for changes in all available socioeconomic and 
demographic variables as well as smoking prevalence proxy variables (Model 3), a 10 pg/m^ 
decrease in PM2,5 was associated with an estimated mean increase in life expectancy of 0,35 
years (SE= 0.l6years,p = 0.033). The estimated effect ofPM2.5 on life expectancy was 
consistent across models adjusting for various patterns of potentially confounding variables 
(e.g. Models 2 - 4). Models 5-9 of Table 2 show the results for select stratified and 
weighted regressions. In counties with a population density greater than 200 people per 
square mile, a 10 pg/m^ decrease in PM2.5 was associated with an increased life expectancy 
of 0.72 (0.22 years, p< 0.01) (Model 6), compared with -0.31 years (0.22 years, p = 0.165) 
in counties with less than 200 people per square mile (P difference <0.01). In counties 
whose proportion of urban residences was greater than 90 percent, a 10 pg/m-^ decrease in 
PM2.5 was associated with an increased life expectancy of 0.95 (0.31, p< 0.01)(Model 7), 
compared with -0.16 (0.16 years, p = 0.299) in counties with less than 90% urban 
residences (P difference < 0.01). 
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When we re-estimated Model 3 of Table 2 using the square root of population density as the 
weight (Model 8 ), the estimated effect of a lOfig/m-^ reduction of PM 2,5 on life expectancy 
was more than double that observed in our un-weighted analysis (0.74 [0.24] vs. 0.35 
[0. 1 6]). When that same model was weighted by the inverse of county land area (Model 9), 
the effect was nearly triple that of the un-weighted analysis (0.96 [0.27]). Additional details 
regarding stratified and weighted analyses are provided in eTables 3 and 4 of the eAppendix. 

We conducted similar analyses for the 2 1 l-county dataset for 1 980 to 2007 and from 2000 
to 2007, the results of which are presented in eTables 2a and 2b of the eAppendix, 
respectively. Results for the period from 1980 to 2000 were identical to those reported by 
Pope et al.^^ 

Figure 3 summarizes the point estimate and 95% confidence interval for the effect of a 1 0 
pg/m^ decrease in PM 2.3 on life expectancy for a select un-weighted and un-stratified 
regression model in each dataset/time period. Models fitted using Datasets 2 and 3 (left) 
controlled for changes in income, population, proportion of the population that is black, lung 
cancer death rate, and COPD death rate, corresponding to Model 4 in eTables 2a,b. Models 
fitted using Dataset 1 controlled for all available variables and correspond to Model 3 in 
Table 2. These estimates were fairly consistent, though estimates corresponding to the 
counties from Pope et aP^ for the period 2000 to 2007 appeared slightly larger than those 
from other analyses. 

In the analyses stratified by sex, the estimated effect of a lOpg/m^ reduction in PM 2 ,s for the 
covariate pattern corresponding to Model 3 of Table 2 was an additional 0.59 (0. 1 7) years of 
life expectancy for women and 0.08 (0.20) years for men (P difference = 0.027). Differences 
by sex were also observed in stratified and weighted models, although with less precision. 
Sex differences were smaller in the most urban counties (urban rate > 90®-o). Similar results 
were observed for the period 1980 to 2000 in Dataset 2. (Sex-specific results are presented 
in eTable 5.) 

Effect estimates were not highly sensitive to the inclusion of the estimated change in 
smoking prevalence. Table 3 summarizes the results for the inclusion/exclusion of the 
smoking prevalence variable across several models. For example, when Model 3 in Table 2 
was re-estimated for the 383 counties with matching smoking prevalence data, a reduction 
of 1 Opg/m-^ was associated with an increase in life expectancy of 0.49 (0. 1 9) years without 
including change in smoking prevalence in the model, and 0.47 (0.19) when including those 
changes. Similar results for smoking were observed in our stratified and weighted models, as 
well as in our models for men and women separately. 

DISCUSSION 

Data on air pollution and life expectancy from 545 U.S. counties in 2000 and 2007 show 
that recent declines in PM 2,5 to relatively low levels continue to prolong life expectancy in 
the US, These benefits are largest among the most urban and densely populated counties. 
These associations were estimated controlling for socioeconomic and demographic variables 
as well proxy variables for and direct measures of smoking prevalence. 

In previous studies, a 10 decrease in PM 2,5 has been associated with gains from 0.42 
to 1.51 years of life expectancy Here, a decrease of I0pg,W in PM 2,5 was associated 
with an increase in life expectancy of 0.35 (0,16) for 545 counties for the period from 2000 
to 2007. An increase in life expectancy of 0.56 (0. 1 9) was estimated for the same 2 1 1 
counties included in the Pope et analysis but extended to the period 1980 to 2007. The 
estimated effect in those 2 1 J counties from 2000 to 2007 was equal to 1 .00 (0.32). Stratified 
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and weighted analyses within the 545 counties from 2000 to 2007 yielded larger estimates 
between 0.72(0.22) and 1.12(0.32) -broadly in agreement with those previously reported. 

From 2000 to 2007, the average increase in life expectancy across the counties in this study 
w'as 0.84 years, and the average decrease in PM 2.5 in those same counties was 1 .56pg/m^. 
While PM 2.5 reductions presumably account for some of the improvements in life 
expectancy over this period, it is only one of many contributing factors. Other factors may 
include improvements in the prevention and control of the chronic diseases of adulthood, 
particularly cardiovascular diseases (CVD) and stroke, and changes in the risk factors 
associated with them, including medical advances, declines in smoking, and decreases in 
blood pressure and cholesterol.^^ Given the well-established link between air pollution and 
CVD mortality,^'^"® and changes in other CVD risk factors, issues of multicausaiity and 
competing risk make it difficult to quantity exactly the changes in life expectancy 
attributable to reductions in PM 2 . 5 . However, if w'e consider one of our more conservative 
effect estimates (Model 3, Table 2) the 1 .56pg/m^ reduction in PM 2.5 accounts for about 
0.055 years ( 1 .56>:0.035) of additional life expectancy, or roughly 7% of the increase in life 
expectancy. Using the estimate from our most urban counties (Model 7, Table 2), the 
increase in life expectancy attributable to the average reduction in PM 2.5 was 0. 148 years 
( 1 .56>^0.095), or as much as 1 8% of the total increase. 

An interesting aspect of this study was how pronounced the PM 2,5 effect was for the original 
211 counties from 2000 to 2007. Given that they were originally selected simply on the 
availability of matching pollution data, wbat is special about these counties that results in 
larger estimates of the effect of PM 2.5 on life expectancy? The stratified and weighted 
analyses suggest plausible explanations. For instance, the 21 1 counties were all in 
metropolitan areas, and the stratified analyses suggest that the effect of PM 2,5 on life 
expectancy is greatest in the most urban counties. One possible reason is that the 
composition of P.M 2.5 is different in urban areas,^^ causing PM 2,5 to have a larger health 
impact. Another possibility is the “non-metropolitan mortality penalty” - the recent 
phenomenon in which mortality rates are higher in mral compared with urban areas. While 
it is not clear why the mortality gap between metro and non-metro areas has widened, some 
hypotheses include greater improvements in standards of care in metro areas, changes in 
uninsurance rates, changes in disease incidence, and changes in health behaviors. These, 
however, would be valid explanations only if they occurred at different rates in metropolitan 
areas compared with rural areas. If so, then perhaps failure to include variables that captured 
one or more of these differences could explain the different estimates of the effect of PM 2,5 
on life expectancy. 

Alternatively, metropolitan areas are more densely populated than non-metro areas. Our 
models that stratified by population density showed that the effect of PM2.5 on life 
expectancy is greatest in the most densely populated smdy areas (those with a population 
density of at least 200 people per square mile) — possibly suggesting a role for differentia! 
exposure misclassification. That is, in densely populated areas, it is more likely that any two 
people from the same area are exposed to the same level of PM2.5 witli perhaps less 
exposure misclassification. This possibility was supported in our models weighted by the 
square root of population density and the inverse of land area, which placed more weight on 
the most densely populated counties and the smallest counties. In these models the effect of 
a 1 0 pg/m-^ decrea.se in PM2.5 on life expectancy was much larger than the equivalent un- 
weighted analysis. 

Another interesting finding was the difference In the effect of changes in PM 2.5 on men and 
women. Findings in the literature regarding the effects of air pollution by sex for long-term 
exposure have been mixed. Studies using the American Cancer Society and Har\^ard Six- 
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Cities cohorts show no significant difference in pollution-related mortality between men and 
women.*^ Studies using a Medicare cohort have reported different effects by age and 
region, but did not stratify by In a study using the Adventist Health cohort, Chen 

et aP® reported a large effect of PM2.5 on fatal coronary heart disease (CHD) in women but 
no association in men. Similarly, in separate studies, Lipsett et al,^^ using a cohort of 
women (California Teachers’ Study), reported associations betw^een particulate matter and 
cardiovascular mortality, while Puett et using a cohort of men (Male Health 
Professionals), found no association with all-cause mortality or fatal CHD. For our main 
analysis using all 545 counties, we find a larger effect of PM2.5 on women, suggesting that 
reductions in PM2.5 are more beneficial to gains in life expectancy for women. Models fitted 
using data for the period from 1980 - 2000 as in Pope et showed similar results. Future 
work should investigate more dioroughly the possibility of different PM2.5*mortality 
associations for men versus women. 

One factor that appeared to play no role in the PM2.5 and life expectancy relationship, 
however, was baseline PM2.5 level. This is in agreement with the findings by Pope and 
colleagues,^^ and implies that, while we may see differences across levels of population 
density, urban rate, and land area, this is not due to these areas having a higher or lower 
baseline PM2.5 level. Furthermore, this finding suggests that there is no clear threshold 
below which fiirther reductions in PM2.5 levels provide no benefit (eAppendix, eTable 3). 
The fact that our results were not sensitive to the inclusion of direct measures of change in 
smoking prevalence suggests that the estimated gains in life expectancy for a 1 Opg/nP 
reduction in PM2,5 are not a result of confounding due to changes in smoking prevalence. 

Unlike previous cross-sectional analyses,*'’-'^ we were able to estimate the association 
between county-specific temporal changes in PM2.5 levels and county-specific temporal 
changes on life expectancy adjusted by temporal changes in several potential confounding 
factors. By looking at within-county temporal changes, we reduce the potential bias due to 
unmeasured confounding. Further, by estimating clustered robust standard errors at the MSA 
level, we took a conservative approach in accounting for potential spatial correlation 
bebA'cen neighboring counties. 

Our analysis has the strengths of using some of the largest available datasets, and applying 
relatively simple analyses. Additionally, we improved on the original analysis by 
constructing a dataset with PM2.5 measured at the county level, in contrast to the more 
coarse MSA-!evel readings used in previous studies.^’^^ 

The analysis is limited, however, in its ability to control for all potential unmeasured 
confounding. Additionally, in comparing selected years, we do not fully exploit potentially 
informative data between those years. Furthermore, sophisticated analyses of the U.S. 
Medicare population hy Greven et al.-^^ did not observe associations between “local” trends 
in PM2.5 levels and “local” trends in mortality in 814 zip code level locations in the U.S. for 
the period 2000 - 2006. “Local” trends were defined as the difference between monitor- 
specific trends and national trends. The Medicare cohorts, however, consisted only of people 
age 65 and older, whereas our life expectancy calculations integrate over all ages. Also, 
other studies using Medicare based cohorts have found significant associations between 
PM2 5 and overall mortality.^®’^^ Future work is needed to investigate whether these 
differences among studies are due to differences in statistical models, data sources, or 
populations studied. 

It is also worth considering whether life expectancy was the most appropriate outcome to 
consider in our model. Because life expectancies are calculated from age-specific mortality 
rates, perhaps a model with age-specific mortality rates as the outcome w'ould be more 
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appropriate, allowing the age groups most affected by PM 2,5 exposure to be pinpointed 
precisely. 

In summary, our study reports strong evidence of an association between recent further 
reductions in fine-particulate air pollution and improvements in life expectancy in the 
United States, especially in densely populated urban areas. 
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Figure 1. 

Map of U.S. with the 545 counties from Dataset i shaded according to year (A) 2000 and 
(B) year 2007 life expectancies. 
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Figure 2. 

Cross-sectional life expectancies plotted vs PM 2.5 levels for (A) 2000 and (B) 2007 in 
Dataset 1 . The slopes of the regression lines correspond to estimates from the simple model: 
LE = intercept + sIope*PM 2.5 in both the 2000 and 2007 plots. (C) On the left the data are 
plotted as change in life expectancy vs change in PM 2.5 over the period 2000 --2007. The 
regression line corresponds to the simple model ALE ~ intercept + slope* APM 2 , 5 (Model I 
in Table 2). (C) On the right is the added variable plot for PM 2.5 corresponding to Model 3 
in Table 2. 
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Figure 3. 

Point estimates (circles) and 95% confidence intervals (vertical lines) for the effect of a 
lO^ig/'m-^ decrease in PM 2.5 on life expectancy. Estimates A and B were obtained from 
Dataset 3; Estimate C was obtained from Dataset 2. Estimates A, B, and C were adjusted for 
changes in income, population, proportion of the population that is black, lung cancer death 
rate, and COPD death rate (Model 4, eTables 2a,b). Estimates D, E, and F were obtained 
from Dataset 1, adjusted for changes in income, population, proportion of high school 
graduates, proportion of the population that is black, proportion of the population that is 
Hispanic, lung cancer death rate, and COPD death rate (Model 3, Table 2). 
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Table 1 

Summary Characteristics of the 545 Counties Analyzed for the years 2000 and 2007 


Variable 

Life Expcclancy (yr) 

2000 

2007 

Change 

PM:3 (tig/mh 
2000 
2007 

Reduction 

Per Capita Income (in thousands of S) 

2000 

2007 

Change 

Population (in hundreds of thousands) 

2000 

2007 

Change 

High School GraduatiK (prop of population) 
2000 
2007 
Change 

Black Population (pn^ ©fpeipuLition) 

2000 

2007 

Change 

Hispanic Population (prop of population) 
2000 
2007 
Change 

Deaths from Lung Cancer (no./IO.OOO pop.)* 
2000 
2007 
Change 

Deaths from COPD (no,/IO.OOO pop.)* 

2000 

2007 

Change 


Mean (SD) 

76.7(1.7) 

77.5 (2.0) 

0.8 ( 0 . 6 ) 

13.2(3.4) 
11 . 6 ( 2 . 8 ) 
1.6 (1.5) 

27.9(7.4) 
30.4 (7.9) 

2.5 (2.3) 

3.5 (6.3) 
3.8 (6.6) 
0.3 (0.6) 

0.81 (0.07) 
0.85 (0.06) 
0.04(0.02) 


0.115(0.1.38) 
0.1!7(0.!39) 
0.002 (0.017) 

0.119(0,189) 
0.098(0.135) 
-0.02! (0.057) 

16.4 (3.5) 
15.5(3.8) 
-0.9(22) 

12.8(3.1) 

12.5 (3.5) 
-0.3(21) 


2005 death rates 


used as a proxy for 2007 death rates. COPD denotes chronic obstructive pulmonary disease. 
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Table 3 

Comparison of results of select models for inclusion of smoking variable vs. no inclusion of smoking variable 


Selected counties 
and analysis 

No. 

Counties 

Full model, with 
smoking 

P(SE, p) for 10 
pg/m^ PM2.5 

Full model, no 
smoking 
p(SE, p) for 10 
Mg/m’ PM,.s 

Ail Counties 

383 

0-47(0.19, 0.01.3) 

0,49(0.19, O.OH) 

2000 population density (persons per square mile) 




>SOO 

no 

0-52(0-43. 0,230) 

0.53(0.43. 0.22!) 

>600 

139 

0.68(0,30. 0.028) 

0.68(0.30, 0.027) 

>400 

187 

0,7!(0,26. 0,007) 

0.70(0,25, 0.007) 

>200 

272 

0,67(0.22, 0.00.3) 

0,65(0.22, 0,004) 

<200 

ni 

'0.50(0.30,0.100) 

-0.39(0,30, 0.193) 

2000 urban rate 




>90% 

157 

0.76(0.28, 0.009) 

0.76(0.28. 0.008) 

>95% 

!0I 

1.01(0,31.0.002) 

0.98(0.32. 0.003) 

<90% 

226 

-O.M{0.20. 0.483) 

-0.13(0.20,0.513) 

2000 population density & 2000 urban rate 




>200 & >90% 

100 

0.95(0.32. 0.004) 

0.9.3(0.32. 0.005) 

Regression weighted by square root of 2000 population density (Ail counties! 

383 

0.77(0.24. 0.002) 

0.76(0.25, 0.003) 

Regression weighted by inverse of county land area (All counties) 

383 

0.81(0.26. 0.002) 

0,74(0.27. 0.007) 

Sex 




Men 

383 

0.20(0.23. 0.389) 

0.22(0.23, 0.343) 

Women 

383 

0.71(0,20, 0.001) 

0.72(0.20, <0.001) 


Vovuriatcs include change in income, change in populaiion, change in high-cchoot graduates, change in proportion of black population, change in 
proportion of Hispanic population, change in lung cancer moftalii>' rate, change tnCOPD mortality rate. Analysis used: SAS 9.2, PROC 
SURVEYREG, clustered by MSA. using the “weight” statement, and ST.AT.A 1 1.0. REGRESS. 
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Responses by Mr. Raymond Keating 

U.S. House of Representative 
Committee on Science, Space and Technology 
Subcommittee on Environment 

Hearing Questions for the Record 
The Honorable David Schweikert 

Ensuring Open Science at EPA 

Answers from Raymond J. Keating 
Chief Economist 

Small Business & Entrepreneurship 


1 . In 2012, the President’s Science Advisor testified that “Absolutely, the data on which 
regulatory decisions... are based should be made available to the Committee and should 
be made public.” Also in 2012, the Chair of EPA’s Science Advisory Board testified that 
EPA’s advisors recommend “that literature and data used by EPA be peer-reviewed and 
made available to the public.” Do you agree with these statements? 

Keating; Yes, I completely agree. When making public policy, it is critical that the process 
be open and transparent, and that includes the science, models and analysis used to formulate 
such policy. 

2. You testified about the disproportionate impact of federal environmental regulations for 
small business and small manufacturers, with cost between 100 percent and 400 percent 
higher than their large counterparts. How does a non-transparent regulatory process 
particularly hit smaller business? 

Keating: The lack of transparency in the regulatory process increases the likelihood that 
regulations are going to be imposed based on special-interest influences and political 
preferences rather than according to rigorous, accurate assessments of actual risks, costs and 
benefits. Regulatory activity based on rent seeking (that is, individuals or groups using 
government to take wealth from others) and special interests (such as those incentivized to 
spend heavily in favor of regulation from which they accrue clear benefits, while the wider 
public, such as consumer or taxpayers, have few incentives to expend the dollars and time to 
counter the issue given the dispersed nature of the costs) is a constant threat. Non- 
transparency merely serves to enable the imposition of misguided regulation, and the costs of 
such regulatory activity falls much more heavily on small firms, as opposed to large 
businesses. 

As a reminder from my written testimony: “The Small Business Administration’s Office of 
Advocacy periodically estimates regulatory costs, obviously with an eye towards the burdens 
imposed on smaller businesses. In September 2010, the Office of Advocacy published an 
updated study estimating the costs of complying with federal regulations. The study - ‘The 
Impact of Regulatory Costs on Small Firms’ by Nicole V. Crain and W. Mark Crain from 
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Lafayette College - provided details regarding the burdens of federal regulatory costs. . . On 
the environmental front, per employee regulatory costs for firms with less than 20 employees 
came in at $4,101, which topped the $1,294 cost for firms with 20-499 employees by 217% 
and the $883 cost for businesses with 500 or more workers by 364%.” 

Clearly, excessive regulation resulting from a non-transparent regulatory process not only 
means increased costs, but also disproportionate burdens on smaller firms. 

3. Can you discuss the relationship between the Secret Science Reform Act and existing 
Administration policy on scientific integrity and transparency, including the President’s 
Executive Order on “Improving Regulation?” 

Keating: Interestingly, the Secret Science Reform Act and President Obama’s Executive 
Order on “Improving Regulation” appear to be mutually reinforcing. In his Executive Order, 
the President said, “Our regulatory system ... must be based on the best available science. It 
must allow for public participation and an open exchange of ideas. It must promote 
predictability and reduce uncertainty. It must identify and use the best, most innovative, and 
least burdensome tools for achieving regulatory ends. It must take into account benefits and 
costs, both quantitative and qualitative. It must ensure that regulations are accessible, 
consistent, written in plain language, and easy to understand. It must measure, and seek to 
improve, the actual results of regulatory requirements.” 

That clearly is reinforced by the Secret Science Reform Act, which requires that the EPA 
"shall not propose, finalize, or disseminate a covered action unless all scientific and technical 
information relied on to support such covered action” is “specifically identified” and 
“publically available in a manner that is sufficient for independent analysis and substantial 
reproduction of research results.” This legislation clearly aligns with the President’s order to 
"allow for public participation and an open exchange of ideas,” “promote predictability and 
reduce uncertainty,” “identify and use the best, most innovative, and least burdensome tools 
for achieving regulatory ends,” “take into account benefits and costs, both quantitative and 
qualitative”, and “ensure that regulations are accessible, consistent, written in plain language, 
and easy to understand.” 

Based on this Executive Order, one would expect the Obama administration to be a strong 
supporter of the Secret Science Reform Act. 

4. EPA also develops and justifies regulations on the basis of economic and air quality 
models that are often not transparent. For example, EPA’s Cross-State Air Pollution Rule 
and its recent New Source Performance Standard for carbon dioxide were developed 
through the use of taxpayer-funded, but proprietary model called the “Integrated Planning 
Model.” 

a. Do you support ensuring that there is public access to these critical models? 
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Keating: Quite simply, any model used to formulate public policy - especially those 
developed with taxpayer funding- must be fully accessible to the public for full 
assessment and review. 

b. As an economist, how important is it to see the inputs, assumptions, and defaults 
within a model in order to assess its quality. 

Keating: Very early in my career, including during graduate school, two big issues 
became clear to me about economic or econometric models. First, models could be 
manipulated to achieve certain results based on the inputs and assumptions used, as well 
as by those left out. Second, models were limited by a host of factors that were critical, 
even central to economic growth, for example, yet could not be quantified in an 
intelligible way to be used in any kind of statistical models. Therefore, it is crucial to see 
the inputs and assumption underlying any scientific model in order to assess how closely 
it tracks to the real world, if you will, and how dispassionate or independent the model is. 

5. In what ways has the development and use of the "social costs of carbon” been 
inconsistent with the Administration’s commitments to transparency? 

Keating: In President Obama’s Executive Order on "Improving Regulation,” it W'as 
clearly stated: “Our regulatory system ... must be based on the best available science.” A 
variety of critics - including, as noted in my written remarks, Robert Pindyck, an 
economics professor at the Massachusetts Institute of Technology - has made clear that 
the “science” upon which the “social cost of carbon” estimates are based has little 
relation to “the best available science.” As noted in the abstract of Pindyck’s paper : “A 
plethora of integrated assessment models (lAMs) have been constructed and used to 
estimate the social cost of carbon (SCC) and evaluate alternative abatement policies. 
These models have crucial flaws that make them close to useless as tools for policy 
analysis: certain inputs (e.g. the discount rate) are arbitrary, but have huge effects on the 
SCC estimates the models produce; the models’ descriptions of the impact of climate 
change are completely ad hoc, with no theoretical or empirical foundation; and the 
models can tell us nothing about the most important driver of the SCC, the possibility of 
a catastrophic climate outcome. lAM-based analyses of climate policy create a perception 
of knowledge and precision, but that perception is illusory and misleading.” 

For good measure, there is the matter in which the Obama administration in effect quietly 
leaked out changes in its “social cost of carbon” guess-timate. As reported in a November 
4, 2013, Bloomberg report . “The administration first set a comprehensive price in 2010, 
and raised it in May of this year after the economic models it used to set the price 
changed to account for rising seas and other natural changes. That change in May was 
slipped into a Department of Energy regulation of microwave ovens, a move criticized by 
Republicans in Congress.” 

None of this speaks to a true commitment to transparency and the best available science. 
Rather, it indicates a regulatory process beholden to a biased, special-interest, and 
political agenda. 
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6. A poll last year found that 90 percent of Americans agree with the statement that studies 
and data used to make federal government decisions should be made public. Do you think 
small businesses agree with this principle? 

Keating: It’s encouraging that the general public polled so high in favor of transparency 
when it comes to governmental decision-making, including regulation. There’s no doubt in 
my mind that small business owners overwhelmingly agree with such sentiments, as they are 
on the frontlines wrestling with the costs and implementation of regulation. 

For good measure, the importance of such transparency for entrepreneurship and small 
business is widely recognized. For example, in the World Banks’s Doim Business 2014 , it’s 
noted: “Regulations that protect consumers, shareholders and the public without over- 
burdening firms help create an environment where the private sector can thrive. Sound 
business regulation requires both efficient procedures and strong institutions that establish 
transparent and enforceable rules.” And a bit later: “Where regulation is burdensome and 
competition limited, success tends to depend on whom one knows. But where regulation is 
transparent, efficient and implemented in a simple way, it becomes easier for aspiring 
entrepreneurs to compete on an equal footing and to innovate and expand.” 

7. During the hearing, you suggested that increases in regulations result in higher costs to 
consumers and businesses. Can you provide examples that are particularly erroneous for 
small business? 

Keating: When it comes to the “social cost of carbon” regulatory scenario, it’s vital to keep 
in mind that government regulating to limit carbon emissions will necessarily drive up the 
costs of carbon-based energy, including coal-produced electricity. As noted in a 2012 report 
on the impact of energy costs on business, 1 highlighted just a couple of examples: 

Indeed, electricity costs often rank among a firm’s biggest expenses. For example, 

Bob Farber, president of Quality Perforating Inc., a Carbondale manufacturer of 
pierced coils, sheets and components, in Scranton, PA, was quoted in a May 25, 

2010, article in the Scranton Times-Tribune observing, “For a business like ours, 
electricity is probably our biggest fixed cost because all our machines are 
electric.” ... 

The particular resources used to generate electricity also will have a clear impact 
on the cost of electricity. Consider the following from a May 8, 2012, 

International Business Times article written by Todd Westby, who is a small 
business owner; “Electricity prices are a big concern for me. And on a tight 
budget, 1 can only account for so much to go toward the electricity bill before 1 
have to pass this cost onto my customers. Luckily for me, 1 live in a state where 
more than 70 percent of the energy is powered by coal. Nebraska's dependence on 
coal has had tremendous results that businesses and families can appreciate. Our 
state has the 1 1th lowest electricity prices in the U.S.... And as I look at coal 
pricing over the last few decades, I've seen one very important fact: it's stable. 
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There is little fluctuation in the already affordable cost of coal from year to year. 

In our uncertain economic times, confidence in low-cost electricity is one fewer 
obstacle to overcome on the road to recovery.” 

For good measure, government mandates, regulations, subsidies and taxes factor 
into the energy cost equation. 

Thank you for your time and attention. Please let me know if there are additional questions. 
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The following documents were submitted foe the record by 
Representative Suzanne Bonamici 


t AMERICAN 
LUNG 

ASSOCIATION. 

Fighting for Air 



■ ■ 

ATS. 


February 10, 2014 


The Honorable Suzanne Bonamici 
Ranking Member, Subcommittee on Environment 
Committee on Science, Space and Technology 
U.S. House of Representatives 
Washington, DC 20515 

Dear Representative Bonamici: 

We are writing to express our opposition to H.R. 4012 the Secret Science Reform Act of 2014. The 
American Lung Association is the oldest voluntary health organization in the United States. The Lung 
Association's mission is to save lives by improving lung health and preventing lung disease. We achieve 
our mission through research, advocacy and education. The American Thoracic Society is a medical 
professional society dedicated to the prevention, detection, treatment and cure of pulmonary disease, 
critical care illness and sleep-disordered breathing through research, education and advocacy 

Science is the bedrock of sound regulatory decision making. The best science underscores everything 
our organizations do to improve health. We strongly believe in a transparent and open regulatory 
process. A vital element of research is patient confidentiality. Physicians and researchers have earned 
the trust of their patients by steadfastly maintaining patient confidentiaUty. Patient confidentiality Is a 
clear legal and ethical obligation. 

The legislation before the committee will compel the U.S. Environmental Protection Agency to either 
ignore the best science by prohibiting the agency from constdeiii^ peer-reviewed research that is based 
on conffdential patient information or force EPA to publicly release confidential patient information, 
which would violate federal law. This is an untenable outcome that would completely undermine ability 
of the EPA to perform its responsibilities under the Ciean Air Act and myriad other federal laws. The 
legislation will not Improve EPA's actions; rather it will stifle public health protections. 

The kind of Information disclosure envisioned in this legislation exceeds that required by peer-reviewed 
journals. We believe much of the intent ofthis legislation Is already achieved through the current peer- 
review process required by all academic journals. The vast majority of peer-reviewed journals require 
manuscript authors to register any trial using human subjects with ciinicaltrials.gov . This public registry 
collects key information on the study population, research goals and methods that aiiow outside 
reviewers and scientists to either challenge or attempt to reproduce study results. Additionally, the 
peer-review process and publication of results invites the broader scientific community to debate study 
findings. Trial registry and manuscript publications are only part of the process by which scientific 
endeavors operate in a transparent environment. 

Private organizations, public charities, research universities, the National Institutes of Health, the 
Centers for Disease Control and Prevention, the Centers for Medicare and Medicaid Services, the 
Department of Veterans Affairs, corporations and many other entities conduct medical research. Many 
of these organizations compile large longitudinal data sets that track patients over a period of time. 
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These data serve as the basis of many studies that permit epidemiologists to track disease and risk 
factor information for large patient populations. 

The published peer-reviewed information from such data often inform regulatory decision making at the 
EPA and other federal agencies as well as future research. Not only do these data inform regulatory 
action, they help Inform efforts to educate the public about the magnitude of a disease, risk factors and 
steps individuals can take to improve their health. In order for EPA to set the most appropriate 
standards, it must be informed by the best information. 

Understanding the impact of air pollution on human health and the magnitude of harm caused by 
pollution at specific levels helps the agency meet its obligations under the Clean Air Act. Absent these 
data, it is unclear upon what basis the agency could make sound decisions. 

We urge the committee to reject H.R. 4012. 


Sincerely, 


Harold Wimmer Stephen C. Crane, PhD, MPH 

National President & CEO Executive Director 

American Lung Association American Thoracic Society 
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Board of Director 

John Applegate 
Robert Glicbman 
David Hunter 
Thomas McGarity 
Amy Sinden 
Sidney Shapiro 
Rena Steinzor 
Robert R.M.Verchick; 

Advisory Council 

Palrido Bauman 
Frances Beinecke 
Euia Bingham 

W. Thompson Comerford, Jr. 
Sally Greenberg 
John Passacontando 
Henry Woxman 
Robert Weissmon 
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February 11,2014 

The Honorable David Schweikert, Chairman 
The Honorable Suzanne Bonamici, Ranking Member 
House Committee on Science, Space, and Technology 
Subcommittee on the Environment 
Washington, D.C. 20515 

Re: Secret Science Reform Act, H.R. 4012 

Dear Chairman Schweikert and Ranking Member Bonamici, 

We are professors of environmental and administrative law who 
specialize in the agencies' use of science in policymaking. We believe that H.R. 
4012, the “Secret Science Reform Act,” contains a number of significant 
problems that cumulatively threaten to undermine, rather than enhance the 
scientific rigor of EP.A's decision-making. We urge you to reconsider the need 
for the bill. At the very least, the bill should be revised significantly before it is 
considered further by the Committee. 

As drafted. H.R. 4012 suffers, at best, from a dangerous lack of clarity. It 
forbids EPA from relying on scientific and technical information unless that 
information is both "specifically identified" and "publicly available in a manner 
that is sufficient for independent analysis and substantial reproduction of research 
results." Neither of the quoted phrases is self-defining. Thus, as it stands, the bill 
will unnecessarily encourage litigation and could lead to a number of other 
significant problems that we identify below. 

H.R. 4012 threatens to undermine the scientific rigor of EPA ’s 
decision-making while leaving the true “secret science” problem untouched. 
“Secret science” is indeed pervasive in some regulatory programs, yet H.R, 4012 
does nothing to address the most serious problems since it inoculates from its 
reach existing, outdated legal provisions that tolerate the sequestration of 
research. For example, under Section 10(g) of the Federal Insecticide, Fungicide, 
and Rodenticide Act, the public and affected parties are not allowed to view the 
studies underlying EPA’s licensing of pesticides until after the agency’s 
registration decision is concluded, and even then the research is available only to 


Center for Progressive Reform 
455 Massachusetts Ave., NW HI 50-5 J 3 
Washington. DC 20001 


www.progressivereform. org 


(202) 747-0698 
phone/fax 
info(^progressivereform. ors 
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the public under tightly constrained circumstances. ' Even more problematic, as a result of 
aggressive trade secret claims, the research on the safety of more than 17,000 chemicals regulated 
by EPA under the Toxic Substances Control Act is completely insulated from public view by law.^ 
Such impediments to public access undermine independent evaluations of the evidence used by 
EPA in its regulation, yet they remain untouched by the very bill that promises to expose this 
secret science. 

By contrast, H.R. 4012 targets publicly available research, much of which has been 
published in peer reviewed journals, as the area in need of heightened transparency. Even more 
perplexing, the bill tasks EPA — not the researchers -- with the enormous task of amassing the data 
underlying each relevant study. If EPA is unable to summon the resources or time to access this 
underlying information or is otherwise unable to acquire the data, it is apparently prohibited from 
considering the stud(ies) in its regulatory decision. 

This draconian requirement will significantly undermine the scientific integrity of EPA’s 
regulation, rather than enhance it, by placing out of the agency’s consideration relevant and 
material studies when EPA is unable to acquire the underlying information. Such an approach 
also provides the opportunity for strategic games. For example, under H.R. 4012, sponsors who 
learn of adverse effects from their products through internal research could attempt to limit EPA’s 
consideration of their findings simply by denying EPA access to their data. Since the data 
underlying privately-funded research apparently remains the property of private parties, they can 
control how their research is used by EPA as best suits their interests. 

The costs of the requirements in H.R. 4012 are grossly disproportionate to any 
plausible benefits. Before imposing this new requirement on the thousands of science-intensive 
projects at EPA, the proponents of such legislative requirements should consider the costs and 
delays to taxpayers and weigh them against the social benefits. The costs are likely to entail tens 
of millions of dollars of staff time, years of delay per standard, and the possibility that EPA will 
either have to bypass considering relevant studies because they cannot make the data available or 
avoid regulatory action altogether. Consider the bill’s requirement as applied to a typical NAAQS 
science assessment by EPA, for example. In its bibliography for this assessment, EPA cites to 
hundreds of peer reviewed, published studies that it considered.^ H.R. 4012, as we read it, would 
require EPA to make the “materials, data, and associated protocols, computer codes and models, 
and recorded factual materials” underlying each of these hundreds of studies “publicly available in 
a matter that is sufficient for independent analysis and substantial reproduction of research 

* A person seeking access to the studies underlying a pesticide registration must certify that he/she will not share the 
information with manufacturers in other countries. In addition, the pesticide manufacturers must be notified of each 
person who views their information; the information must be viewed at the agency’s office; and the information is 
available only after a pesticide registration decision is made. 7 U.S.C. § 136a(c){2)(A). 

^ See Declassifying Confidentiality Claims to Increase Access to Chemical Information, EPA, 
httD://www.eDa.gov/onDt/existingchemicals/pubs/lransoarencv.charts.html (last visited Apr. 15, 2013) (listing, in 
Table 4, 17,03 1 CBI chemicals on TSCA inventory). 

^ See EPA’s Integrated Science Assessment for Particulate Matter, Dec. 2009, Annex E References, at E-522, 
available at httD://www.epa.gov/ncea/pdfs/Dartmatt/Dec2009/PM ISA full.pdf 
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results.” Indeed, aggressive interest groups could even argue that the bill requires EPA to put the 
raw data into an electronic database to expedite statistical analysis. The costs resulting from such 
a demand on EPA would be extremely high — a wasteful outcome in an era of budget shortages. 


The benefits to regulatory quality - by contrast - seem miniscule. In how many of these 
agency actions will affected groups actually benefit from this enhanced access to underlying data? 
And who are the groups with the resources and interest to reanalyze the data or reproduce the 
study? They certainly are not the groups that are thinly financed. And what is to be gained from 
the resultant reanalysis? Is the agency equipped to review meta-analyses of data bases that have 
not been peer reviewed, published, or restricted methodologically? Indeed, as between peer 
reviewed studies and non-peer reviewed re-analyses of data, is Congress suggesting the latter is 
preferable or even desirable?'* 


Moreover, while we strongly support extending the Data Access Act to private parties, as 
has been suggested by the Bipartisan Policy Center, the Administrative Conference of the U.S., 
and in articles we’ve written, such a requirement should never preclude the agency from using 
studies when the data is not publicly available.^ Imposing such a prohibition on the agencies 
makes “the perfect the enemy of the good” - limiting the agency’s access to scientific research 
based on expensive and often fruitless paperwork requirements. 


H.R. 4012 facilitates further mechanisms for harassing scientists. There have been 
repeated, documented incidents of the harassment of researchers whose results produce unwanted 
results for regulated parties.* In a number of these incidents, the harassing party’s first line of 
attack begins with subpoena-ing or otherwise acquiring the underlying data and then statistically 
reanalyzing the data in ends-oriented ways that attempt to cast doubts on the integrity of the 
researcher.’ H.R. 4012 provides still more tools for disgruntled interests to “manufacture doubt.” 
If Congress seeks to legislate additional opportunities to enable this type of harassment, it should 
also legislate protections for researchers so that our most talented scientists do not leave the health 
and environmental science field altogether. As Dr. Donald Kennedy, the former Editor in Chief of 
Science, warns: 

I know what many of my fellow scientists are saying to one another .... They 
wonder whether the data underlying their findings may be subject to examination 


See generally Sheila Jasanoff, Transparency in Public Science: Purposes, Reasons. Limits, 69 LAW AND 
Contemporary Problems, Summer 2006, at 22 (providing an excellent overview of the dangers of hyper- 
transparency provisions, such as those embodied in the Secret Science Reform Act). 

^ See, e.g.. Bipartisan Policy Center, Improving the Use of Science in Regulatory Policy 43 (Aug. 2009); ACUS 
Recommendation 2013-3; Wendy Wagner and David Michaels, Equal Treatment for Regulatory Science: Extending 
the Controls governing the Quality of Public Research to Private Research, 30 AMERICAN JOURNAL OF LAW & 
MEDICINE 119 (2004). 

* See, e.g., DAVID MICHAELS, DOUBT IS OUR PRODUCT (2008); NAOMI ORESKES & ERIK CONWAY, MERCHANFS OF 
Doubt (2009); Thomas 0. mcGarity & Wendy e. Wagner, bending Science: How Special interests corrupt 
PUBLIC Health Research (2008). 

’ Herbert L. Needleman, Salem Comes to the National Institutes of Health: Notes from Inside the Crucible of 
Scientific Integrity, 90 PEDIATRICS 977 (1992); Paul M. Fischer, Science and Subpoenas: When Do the Courts 
Become Instruments of Manipulation, 59 L. & Conlemp. Problems 159 (1996) (both articles are included as 
attachments to the letter on file with the Subcommittee). 
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and reinterpretation, perhaps with some ‘spin’ supplied by the revisionists. They 
know that charges of research misconduct could arise from hostile access to their 
scientific work. They know they are vulnerable to personal attack from those whose 
interests may be adversely affected by the product of their research.® 

These are only a few of the many problems we have identified with the bill, but given your 
upcoming hearing, we believe it is better to share some of them early in the discussions. We are 
happy to provide a more comprehensive assessment later on, as the legislative drafting progresses. 


Thank you for your consideration. 
Sincerely, 


John S. Applegate 

Waiter W. Foskett Professor of Law 
Indiana University Maurer School of Law 


Holly Doremus 

James H. House and Hiram H. Hurd Professor of 

Environmental Regulation 

University of California Berkley School of Law 



Emily Hammond 

Professor of Law 
Wake Forest University 


Thomas O. McGarity 

Joe R. and Teresa Lozano Long Endowed Chair in 

Administrative Law 

University of Texas School of Law 



Noah M. Sachs 
Professor 

University of Richmond School of Law 



Sidney A. Shapiro 

University Chair in Law 
Wake Forest University 


Rena Steinzor 

Professor of Law 

University of Maryland Francis King Carey 
School of Law 



Wendy Wagner 

Joe A. Worsham Centennial Professor 
University of Texas School of Law 




® Don Kennedy, Prologue, at xxiii, in RESCUING SCIENCE FROM POLITICS (Wendy Wagner & Rena Steinzor eds. 
2006 ). 
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Salem Comes to the National Institutes of Health: Notes From Inside the 
Crucible of Scientific Integrity 


Herbert L. Needleman, MD 


ABBREVIATIONS. NIH, National Institutes of Health; EPA, En- 
vironmental Protection Agency; WISC-R, Wechsler Intelligence 
Scale for Children-Revised. 


Many readers of Pediatrics may have only a dim 
idea of the combative arena in which environmental 
research is conducted. Probably, very few have had 
the experience of being investigated for scientific mis- 
conduct. My aim in reviewing these two topics is to 
provide a preventive road map to others and to reveal 
some inadequacies and inequities in the investigative 
process. It is necessary, to accomplish this, to be direct 
and specific. Tact is sacrificed here for the sake of 
clear instruction. 

In 1972 1 published 700 words in Nature reporting 
that Philadelphia inner-city children had higher den- 
tine lead levels than suburban children.’ The paper 
suggested that the tooth might be a useful marker to 
estimate body lead burden after exposure had ended. 
1 did not know then that 1 was taking the first step 
toward being investigated for scientific misconduct 
by my university and the National Institutes of Health 
(NIH) Office of Scientific Integrity. 

The Environmental Protection Agency (EPA) asked 
me to present the 1972 tooth lead paper in Amster- 
dam at an international meeting on lead. I was un- 
prepared by my past attendance at pediatric meetings 
for what 1 encountered there. This was no scholarly 
debate on the toxicology and epidemiology of lead; 
this was war. The speakers did not behave like aca- 
demics hoping to embellish their reputations by pa- 
rading the results of their last 6 months in the lab. 
Th^ stakes were much higher. 

Arrayed against each other were a small and defen- 
sive group of environmentalists and health scientists 
on one side, and on the other the representatives of 
the gasoline companies, including such formidable 
entities as El DuPont, Associated Octel, Dutch Shell, 
and Ethyl Corporation of America. Any paper sug- 
gesting that lead was toxic at lower doses immediately 
faced a vocal and well-prepared troop that rose in 
concert to attack the speaker. My 10-minute talk was 
not spared; giving it marked the beginning of my 
post-postgraduate education. 

This encounter pushed me, on returning to the 
United States, to look into the history of lead research. 
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I found that my experience was not new. Two Aus- 
tralians, A. J. Turner and J, L. Gibson, who first 
described childhood lead poisoning in Brisbane in 
1892, were derogated by industry and by a segment 
of the medical community. When Randolph Byers, 
one of the earliest pediaMc neurologists, first sug- 
gested in 1943 that some school dysfunction might 
be due to undiagn(»ed lead toxicity, he was threat- 
ened with a million dollar lawsuit by Lead Industries 
Association.* Clair Patterson, the geochemist credited 
with dating the age of the earth, was publicly vilified 
as a crank by the industry and had his career threat- 
ened when he sug^sted that civilization had raised 
everyone's body lead burdens to 1000 times that of 
our ancient ancestors (personal communication, 
1992). A!) of the early research in lead toxicity was 
funded by the industry, who had a tight grip on what 
the public was permitted to know.^ 

Reading these records vividly brought back an ex- 
perience I had when I was in medical school. One 
summer I worked as a laborCT at the Deepwater, NJ, 
CKiPont plant, where tetraethyl lead had been syn- 
thesized years before. Workm were forbidden to 
carry matches, and when ihe smoking whistle blew 
at 10 AM and 2 pm, we poured out of our buildings by 
the hundreds to collect at wooden smoking shacks in 
open areas. There we lined up at two glowing cigar 
lighters imbedded in the shack wall. While I smoked 
two cigarettes back-to-bad( in the IS-minute break, I 
inspected my coworkers. 

Off to the side sat a few older men, obviously slow 
and clumsy, staring sUendy into middle space. When 
they did speak, they seeing remote and out of touch. 
A veteran worker told me that they were from The 
House of Butterflies.* They had been poisoned while 
making tetraethyl lead. Yean later, I would read in 
the American Journal of Public Health that during the 
early stages of tetraethyl lead production at Deep- 
water,^ there had been an outbreak of poisoning 
among the work femre. More than 300 men had been 
affected, often with full-blown psychotic symptoms; 
at least 4 had died. Affected workers were frequently 
%en brushing hallucinated insects off their bodies, 
hence the name. Production was temporarily stopped 
by the Public Health Service, but this ban was Ufted 
after a superficial investigation. These damaged men 
were some of the survivors. 

Years later, having satisfied myself that the tooth 
was a valid marker of past exposure, with Alan Lev- 
iton and Bob Reed, I studied a sample of children 
who were asymptomatic for lead, classifying them by 
dentine lead levels. The data showed that after con- 
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trolling for a number of covariates, children with 
elevated lead in their teeth scored lower on tests of 
psychometric IQ, speech and language function, and 
on measures of attention.® The study seemed to re- 
spond to a number of research difficulties that had 
until then vexed the field, and as a result it received 
considerable attention. The lead industry, in the form 
of the International Lead Zinc Research Organization, 
was uncharacteristically silent for about 6 months. 
Then they began to call for copies of my original data. 
1 declined. 1 had seen what had happened to good 
data when massaged and distorted by industry tech- 
nicians, and while 1 was happy to share my data with 
any bona fide scientist — and did — I was not willing 
to include the lead industry. 

In 1982, the EPA began to rewrite the Air Lead 
Standard. I was asked to participate. Also invited was 
Dr Claire Emhart, a psychologist who had published 
a paper in 1974^ that reported that lead was associated 
with lower IQ in a group of Long Island black pre- 
schoolers. In 1981, she published a paper (in this 
journal) which criticized my study and said that when 
followed into the first grade, the lead effect she had 
previously reported was no longer significant.^ Close 
examination of the paper showed that school-age 
blood lead levels were in fact significantly related to 
IQ. Emhart dismissed this finding as due to chance, 
and stated that: 'If there are, in fact, behavioral and 
intellectual sequelae of low levels of lead bur- 
den. . . these effects are minimal.' Shortly after that 
paper she became a grantee of the International Lead 
Zinc Research Organization and began to speak 
against controlling lead in the environment. VVT\en 
there was a move to put lead back in gasoline, Emhart 
appeared in testimony for Lead Industry Associates, 
asserting that there was no valid health reason to ban 
its use.^ 

The industry began to raise public questions about 
the integrity of my studies. In 1983, EPA's Clean Air 
Scientific Advisory Committee thoroughly reviewed 
industry-generated charges that my work was flawed. 
They concluded: 

A pioneering general population study was reported by Needle- 
man et a! (1979). . . . Significant effects (p < -05) were reported for 
full scale WISC-R [Wechsler Intelligence Scale for Chiidren-Re- 
vised] scores, WISC*R v«bal IQ scores, for 9 of II classroom 
behavioral scale items, and several experimental measures of per- 
ceptual motor function. . . . 

Reanalyses carried out in response to the Committee's recom- 
mendaticms have been reported by Needleman (1984). Needleman 
et al (1985) and US EPA's Office of Policy Analysis (1984) as 
connrming the published findings on significant associations be- 
tween elevated dentine lead levels and decrements in IQ. . . ’ 

I thought that this official statement had finally and 
permanently sealed the argument. I could have not 
conceived that these same charges would be resusci- 
tated 7 yeare later. 

In 1990, an attorney from the Department of Justice 
asked me to participate in what he described to me 
as a landmark suit brought under the Superfund Act 
against three lead polluters in Midvale, UT. Among 
the witnesses for the defense were Dr Emhart and Dr 
Sandra Scan. Scarr had been a member of the gov- 
ernment committee that had reviewed my work for 
EPA. She now appeared in a different role, this time 


on behalf of the lead industry, reviving the same 
charges that had been settled in 1986. They came to 
my lab for 2 days to examine my raw data in prepa- 
ration for the trial. 

Before going to trial, the case was settled. Sixty- 
three million dollars was awarded to the federal gov- 
ernment to clean up the mine site. After the case was 
settled, 1 found out that Scan and Emhart had written 
a lengthy document accusing me of unscientific be- 
havior. They maintained that their conclusions grew 
out of their examination of my printouts. This docu- 
ment was forwarded to NlH's office of scientific 
misconduct by David Genesson, an attorney for the 
Washington, DC, law firm of Hunton and Williams. 
It was also given to defense lawyers in a number of 
lead damage cases. 1 had encountered the name of 
Hunton and Williams before. This firm had repre- 
sented Ethyl Corporation of America and El DuPont, 
contesting the regulation of lead additives in federal 
court and before the EPA and the Federal Trade 
Commission. In reading the Scarr/Emhart document, 
I found numerous allegations and hints of unscientific 
behavior. 

As I perceived them, their major criticisms of my 
work were (1) that 1 did not properly control for 
confounding; (2) that I selected cases in a biased 
fashion; and (3) that multiple tests were done, and 
this could lead to positive associations on the basis of 
chance. 

These kinds of issues are generally considered 
methodological disagreements and are fought out in 
the pages of journals; I could not understand why 
they were defined by my critics as scientific miscon- 
duct. Similar criticisms were raised before the EPA in 
1982 and dismissed. These facts notwithstanding, in 
October of 1991, 1 was notified by the Dean of my 
medical school that an inquiry into charges of mis- 
conduct was being done at the instruction of NlH's 
Office of Scientific Integrity. 

When the proceedings began, I was confident that 
the printouts would be examined, that 1 would explain 
how 1 analyzed the data, and that like the EPA, the 
university would rapidly put matters right. I thought 
this would end this matter quickly and permanently. 
But the university's behavior seemed odd and trou- 
bling. They chose to ignore a number of rather ob- 
vious facts that I repeatedly brought to their attention: 
that the charges were initially raised by two individ- 
uals who had been supported by the lead industry; 
that they had been raised before and dismissed by 
the Clean Air Science Advisory Committee of the 
EPA; that my work had been replicated more than 12 
times since its publication; and that I had shared my 
data with other scientists in the past. 

Instead, the preliminary Inquiry Panel issued a 
strange report. The Panel stated that it 'found no 
evidence of fraud, falsification or plagiarism,' but 
inexplicably added that it 'is not able at this time to 
ecdude the possibility rule of scientific misconduct in 
terms of misrepresentation.' The report argued that 
the models I chose were selected to optimize a lead 
effect, and that I may have selected cases in a biased 
fashion. The report presented no evidence in support 
of this assertion, only conjecture. 
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1 rebutted their charges in a letter to the Dean and 
showed that the charge of misrepresentation was 
based on false evidence. The Dean declined to review 
my letter. Instead, he turned it over to the Panel for 
comment. They also did not respond to any of the 
facts that I raised in the letter. Instead, they stated 
that the material 1 supplied in rebuttal of the report 
of the Inquiry Panel was 'not directly relevant,' They 
recommended a full investigation. 

During the time the investigation was being ar- 
ranged, I requested of the Dean that the Hearing 
Board he appointed include experts of international 
standing in the fields of behavioral toxicology and 
epidemiology. This was denied. 1 was told that there 
was no ne^ for this expertise in the two disciplines 
that my work spanned. 1 requested that the hearings 
be open to the university community and the press. 
Again, this was denied. I asked that two members of 
the Hearing Board be replaced for possible conflict of 
interest. One, Dr Robert McCall, was a developmental 
psychologist whose appointments on many profes- 
sional committees overlapped with Dr Scarr, and who 
frequently cited her work in support of his. The 
second. Dr Herbert Rosenkranz. had been Director of 
the Environmental Sciences Center at Case Western 
Reserve University, where Dr Emhart was a faculty 
member. This request was also denied. 

1 began to feel uneasy and increasingly certain that 
if the case were review^ in amen, ! would be found 
guilty of something. 1 went before the Faculty Assem- 
bly of the university and requested their support in 
my demand for open hearing^. The faculty emphati- 
cally supported me. The Assembly passed a unani- 
mous resolution asking the university to open the 
hearings. At the Faculty Senate, a representative of 
the administration argu^ agairtst open hearings, be- 
cause, he said, it was necessary to 'protect the proc- 
ess.' The 'need to protect the process' was a phrase 1 
was to hear repeated many more times. 1 argued that 
the process did not have a nervous system; that it 
was people who required protection; and that the 
given reason that hearings were closed was to protect 
the reputation of the accused. I was in this instance 
the accused, and I wanted ttw hearings to be open. 

The Senate unanimously voted for open hearings. 

Pressure began to build on the administration, and 
1 began to receive letters of support from colleagues 
around the country. Six eminent health scientists, 
Frank Oski, Arthur Upton, Samuel Epstein, Philip 
Landrigan, David Bellinger, and Bernard Weiss spon- 
sored a petition to the Chancellor demanding open 
hearings. It listed almost 400 scientists' signatures. 1 
filed a complaint in federal court asking for open 
hearings. Reluctantly, for the first time in its history, 
the university agreed to open hearings. 

My accusers, who until then had been quite public 
and emphatic in their allegations, and who had said 
that they would willingly come to Pittsburgh to be 
questioned by me, reversed their field. They were 
now reluctant to attend. After lengthy negotiations 
with the administration, they agreed to attend the 
hearings as witnesses. 

The hearing room was filled with scientists, faculty, 
and members of the local and national press. My 


accusers became surprisingly reticent. Dr Scarr, in a 
lecture at the Massachusetts Mental Health Center,^” 
said: 'What we have done is to report, . . Dr Needle- 
man to the Office of Scientific Integrity at the NIH, 
because we feel there are significant deviations from 
normal scientific practice here and we feel that the 
data has been massaged, to put it mildly. . . Now, 
in an open hearing, she revised her complaint to say 
that she merely 'had suspicions' that 1 had con- 
sciously manipulated the data to present a false case. 

Both witnesses were accompanied by their attor- 
ney, Mr David Genesson of Hunton and Williams of 
Washington, DC. When I asked Dr Emhart who was 
paying her legal bills, she refused to answer. She 
stated that she did not know that Hunton and Wil- 
liams had represented El DuPont and Ethyl Corpo- 
ration of America before the Food and E>rug Admin- 
istration and Federal Trade Commission. In the news- 
paper the next day, it was reported that there was a 
'trust fund' established to cover my accusers' legal 
expenses, but that Scarr and Emhart did not know 
who had contributed to it. 

During my examination of my accusers, it became 
clear that a different standard, perhaps an ad hoc 
standard, was being applied to my work as contrasted 
to theirs. One of the charges raised by my accusers 
was that I did not control for age in evaluating the 
effect of lead on IQ. 1 pointed out in my cross- 
examination that the WISC-R IQ was age-adjusted. 
DR NEEDLEMAN: isn't the Weehster age adjusted? 

DR ERNHART: The nonning of the Wechsler is age ad- 
justed. . . norming alone is not sufficient to handle age variation. . . 
DR NEEDLEMAN: So it would be better to enter age into the 
model? 

DR ERNHART: Yes. . , 

DR NEEDLEMAN; In your 1981 paper did you put age into the 
model? 

DR ERNHART; My study is ineievani to the issues here today." 
(Emhart had not controlled for age.) 

Since Emhart had raised these criticisms of my 
work in 1981, and examined my printouts in 1990, 1 
asked her whether it was not true that she had 
concluded that my study misrepresented the data 
before she had ever examined my data. Her answer 
was intriguing. 

DR ERNHART: On advice of counsel. I'm not answering that 
question. 

Another claim was that I excluded subjects on the 
basis of head injury or history of exposure or being 
non-English speaking after I knew their IQ scores, in 
order to maximize the effect of lead. In the hearing I 
showed her a piece of computer code from by printout 
that headed every data analysis. Translated, it said: 
'Select if lead level equal high or low, and head injury 
equal 'no,' and plumbism equal 'no' and English is 
the first and only language in the home.' This proved 
conclusively that the subjects were excluded on cri- 
tCTia that were identified before the study was begun, 
and that the exclusion was executed by computer 
without any human judgment. Because Dr Emhart 
had spent 2 days with my printouts as part of the 
Midvale suit, 1 asked whether she had seen this piece 
of code. 
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DR NEEDLEMAN; .... does this look familiar to you at all? 

DR ERNHART: i don't recall having seen this. 

This piece of computer code appeared 24 times in 
the printouts I furnished them. It is difficult to see 
how it could have been overlooked by anyone looking 
for problems in case selection. 

liiere was a general retreat by both witness in the 
degree of certainty with which they indicted me. 
Scarr, who had been direct and accusatory in a lecture 
at Harvard, was much less sure about whether 1 
committed scientific misconduct in the public hearing. 
1 asked her about it directly: 

DR NEEDLEMAN: Are you certain that you are right when you 
say I selected the cases consciously knowing the outcome in relation 
to lead? 

DR SCARR: ! know you had the opportunity to do that. 1 don't 
know what you did. 

At the conclusion of the CToss-examination, Dr 
William Cooley, Chairman of the Hearing Board, who 
had frequently advised my accusers that they were 
not required to answer my questions, addressed him- 
self to Dr Scarr: 

I believe that, if I may ask a clarifying question, it is my 
impression that you have gone on record here today as essentially 
in^cating that you had ample basis for being suspicious of the 
sdendfic work that's under consderation here, but have no specific 
charges of misconduct. 

DR SCARR: Yes, that’s correct. 

The 2-day hearings were widely reported in the lay 
press’^’’^ and in Science^* and the Journal of the Na- 
tional Institutes of Health. Two months later, on May 
20, 1992, the Hearing Board unanimously found no 
evidence of scientific misconduct. 

What is there to be learned from this story? 1 believe 
that the spectrum of those behaviors labeled as mis- 
conduct in scientific enterprises is disturbingly com- 
mon and that both the public and the scientific en- 
terprise needs to be protected from inferior or dishon- 
est studies that o^n the door to procedures or 
pharmaceuticals of dubious efficacy or that distort 
our understanding of the way that nature works. I 
believe that because of the intensely competitive busi- 
ng that science has become, the ethos in which 
young scientists are socialized and the actual work is 
conducted has fundamentally changed, and not for 
the better. Young scientists are regularly exposed to 
the gap between the professed idealistic standards of 
practice and the actual, often cynical, conduct of grant 
getting, data collecting, interpreting, and publishing. 
There needs to be better policing of our profession. 

But the entire tangled process of identifying puta- 
tive cases of scientific misconduct, and of fairly judg- 
ing them, is open to abuse at a number of points. If 
my case illuminates anything, it shows that the fed- 
eral investigative process can be rather easily ex- 
ploited by commercial interests to cloud the consensus 
about a toxicant's dangers, can slow the regulatory 
pace, can damage an investigator's credibility, and 
can keep him tied up almost to the exclusion of any 
scientific output for long stretches of time, while 
defending himself. 

Some way must be found to screen out frivolous 
or harassing charges of misconduct and shield inves- 
tigators from this form of tribulation. Once an inquiry 


or investigation has begun, it should operate under 
formal principles of due process. The option as. to 
whether the investigation is open should lie with the 
accused. If an open hearing is requested, it should be 
freely granted. One should not be required to fight 
for this long-honored right. Certainly there is stigma 
and embarrassment attached to this charge; th^e are 
trivial compared with the risks that attend closeted 
star-chamber proceedings. One can live with embar- 
rassment. 

The charges should be given in specific written 
form to the accused party. They should take the shape 
of single valued propositions that can be disproven. 
Vague charges of guilt are out of place in a free 
society. The accused should have an attorney of his 
or her choice furnished by the university. The rules 
of evidence and the burden of proof should be clearly 
defined. Full and unhindered cross-examination of 
the accusers should be allowed. Each authority, 
whether university, hospital, or research institute, 
should have an ombudsman group with official, not 
advisory status. At my university, there is a standing 
committee on academic freedom which serves this 
role, but it has little official standing. A majority of 
the members of any investigative panel should be 
constituted from experts outside the university. Full 
disdosure to avoid conflicts should be required. These 
should be chosen in the same fashion as a jury, with 
challenges for cause allowed. 

What can a young investigator do to avoid this 
unpleasantxvess? First, be honest. 1 do not intend this 
to be facetious. Begin by avoiding work that you 
believe is douded by proprietary interests. Avoid 
contract work to fill our your salary or the depart- 
ment's budget. I say this recognizing that this is a 
difficult imperative, particularly for young investiga- 
tors in funding times, but much of this work 

can carry pressure, even if unstated, to find a certain 
effect. Recognize the pressure that accompanies the 
need to produce a publishable study or a given effect. 
Evaluate what the cost to you might be. In choosing 
a mentor, select one whose value system places hon- 
est sdence over publishable results. 

Discuss with your associates steps to take to mini- 
mize bias, consdous or unconscious. Consult a good 
biometrician or epidemiologist about these questions 
early in the planning of the project. Record these 
discussions in a bound book. Remember that years 
later you may be asked to defend your choices of 
methods. Keep your data in two secure places, and 
document the means taken to find, classify and scale 
subjects and any changes in protocol. In a recent 
paper, Freedland and Carney^® polled a group of 
highly regarded investigators and found that a ma- 
jority had trouble recallhig the methods used to clas- 
sify patients. Keep minutes of staff meetings, and 
document discussion of problems. Consult with ex- 
perts in the difficult methodological areas. Ask them 
for written comments. Be skeptical of your conclu- 
rions. Write up and submit negative studies for pub- 
lication. Be modest in your claims. 

Finally, work to reform the system at every level. 
Discuss these issues in research conferences, at insti- 
tutional review board meetings, and at meetings of 
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scientific societies. Do not avoid difficult areas of 
investigation. Take risks. If scientists exclusively 
choose the safe routes, avoid controversial research 
problems, and play only minor variations of someone 
else's themes, they voluntarily turn themselves into 
technicians. Our craft will indeed be in peril. Rnd 
and nurture good colleagues who will insist on the 
best from you, tell you when you are wrong, and 
stand with you in a difficult time. They are truly 
treasures, and their friendship will endure and sustain 
you past all confusion and pain. This article is a deeply 
felt thank you note to the many valued men and 
women who did precisely that for me. 
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Editor's Note 

I asked Dr Needleman to write up his experience 
with the court system and the National Institutes of 
Health Office of Scientific Integrity, I tried to follow 
this case in the press, but 1 didn't find this very 
satisfactory. If you're searching for truth you rarely 
find it in newspapers. Now that I've read Dr Needle- 
man's story 1 have a clearer idea of his ordeal, but I 
am confused. Dr Needleman believes he has been 
found not guilty. The government (Environmental 
Protection Agency) and other scientists also believe 
this, but others may not (see page 978, the preliminary 
report of the Inquiry Panel). 

How long must this go on? Has Dr Needleman 
been victimized over a difference of opinion about 
the quality of his science? 

Editors are exposed daily to conflicting c^rinions. it 
has never occurred to me to take such matters to court 
to be settled! Conflicting opinions are conuncm and 
very important in science. Truth doesn't emerge eas- 
ily. Many studies are often needed before one side 
convinces the other that they are right Sdentiftc 
debates can't be settled in courts! 

I expect that we will hear the opinions and view- 
points of others about this in our Letters to the Editor 
column in the next issue of Pediatrics- 

J. F. L, MD 


WRITING AND THINKING 


As anyone knows who has ever sat down to write, writing is thinking. The 
thought not only precedes the word, it follows it too: we do not know what we 
mean to say until, after many trials and errors, we have found the words. The 
purpose of writing well is thinking well. 


Raimi j. By any other name. Columbia Magazine. 1991; {Spring):36-37. 
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SCIENCE AND SUBPOENAS: WHEN DO 
THE COURTS BECOME INSTRUMENTS 
OF MANIPULATION? 


Paul M. Fischer, M.D.* 

I 

Introduction 

On December 11, 1991, the Journal of the American Medical Association 
(‘JAMA ") published three studies that examined the effect of the Camel ciga- 
rette “Old Joe” advertising campaign on adolescents and children.' I was lead 
author on the study that showed that “Old Joe” was nearly universally recog- 
nized by six-year-old children, a level of awareness that matched the logo for 
the Disney channel. Because cigarette smoking is the leading preventable 
cause of death and disease in this country, I recognized that this research might 
play a prominent role in the subsequent debate about tobacco advertising. As 
a scientist, I naively assumed that this discourse would be conducted in aca- 
demic journals based upon rigorous research and leading to an improved un- 
derstanding of whether and how advertising influences adolescent experimen- 
tation with cigarettes. To date, most of the subsequent debate has occurred in 
court. 

From the beginning, the tobacco industry attempted to discredit this re- 
search and harass the researchers. My experience in confronting the tobacco 
industry has taught me how easily the courts can become the unwitting accom- 
plices of an industry whose goal is profit, not the identification of scientific 
truth. In his paper in this issue of Law and Contemporary Problems, Michael 
Traynor states that with “common sense and goodwill in every quarter” there 
should be few problems due to compelled discovery of scholarly research.^ Un- 
fortunately, in some cases, neither common sense nor goodwill prevail. In such 
cases, the court can become an instrument of abuse. 


Copyright © 1996 by Law and Contemporary Problems 
‘Doctor of Family Medicine, Evans, Georgia. 

1 . J oseph R. DiFranza et at, RJR Nabisco ’s Cartoon Came! Promotes Came! Cigarettes to Chii- 
dren, 266 JAMA 3149 (1991): Paul M. Fischer et al.. Brand Logo Recognition by Cbiidren Aged 3 to 6 
Years: Mickey Mouse and Oidjoe the Camel, 266 JAMA 3145 (1991); John P. Pierce et at, Does To- 
bacco Advertising Target Young Peopie to Start Smoking?, 266 JAMA 3154 (1991). 

2. Michael Traynor, Countering the Excessive Subpoena for Schoiarly Research, 59 Law & 
CONTEMP. PROBS. 119, 148 (Summer 1996). 
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II 

My Introduction to Excessive Subpoenas 
A. A Chronology of Events 

The “Old Joe” studies were published in a JAMA theme issue dealing with 
tobacco research.’ The American Medical Association also held a press con- 
ference in New York to present the findings/ which received wide coverage in 
the press.’ 

On March 9, 1992, The American Medical Association, the Surgeon Gen- 
eral, the American Cancer Society, the American Heart Association, and the 
American Lung Association called for a ban on "Old Joe" advertising attrac- 
tive to children.’ The following day. James Johnson, C.E.O. of the R.J. Rey- 
nolds Tobacco Company (“RJR"). defended “Old Joe" in an interview pub- 
lished on the editorial page of U.S.A. TodayJ In this interview, he attacked the 
“Old Joe” studies and its researchers.’ Mr. Johnson argued that the “studies 
are flawed in very serious ways. The scientists who wrote these studies are not 
unbiased.”’ He made two specific claims about our research that were not true. 
He stated that the sample size was twenty three people'” when in reality it was 
229 people. He also claimed that we called the parents of the three- to six-year- 
old children in our study the night before the data collection and asked them 
only about cigarette use." This statement was a total fabrication. Such a call to 
the parents would have obviously biased the results. 

On March 27, I was served a subpoena duces tecum by RJR." A suit had 
been filed in California by Janet Mangini against RJR, based on RJR’s failure 
to place health warnings on promotional products such as Camel caps and t- 
shirts." I received the subpoena even though my research had not been named 
in the Mangini complaint, I was not a witness to either side in the case, and my 
1991 JAMA research had no bearing on the issue of health warnings. 

The subpoena ordered me to produce the following: the names and tele- 


3. 5ee supra note ! . 

4. See Stuart Elliott, Top Health Official Demands Abolition of “Joe Camel" Ads, N.Y. TIMES, 
Mar. 10, 1992, at A!. 

5. See, e.g., Jane E. Brody, Smoking Amcmg Children Is Linked to Cartoon Camel in Advertise- 
ments, N.Y. Times, Dec. 11, 1991, at D22, 

6. Elliott, supia note 4. 

7. RJ. Reynolds: Ads Do Not Cause Kids to Smoke, U.S.A. TODAY, Mar. 10, 1992, at 9A. 

8. See id. 

9. Id. 

10. See id. 

11. See id. 

12. Defendant’s Notice of Out of State Deposition on Oral Examination and Request for Produc- 
tion of Documents and Things, Mangini v. RJ. Reynolds Tobacco Co., No. 939359 (Super. Ct, San 
Francisco Cty., Cal, Mar. 30, 1992). 

13. See Mangini v. R.J. Reynolds Tobacco Co., 21 Cal.Rptr.Zd 232 (Cal Ct. App.), rev. granted & 
opinion superseded, 859 P.2d 672 (Cal. 1993), cert, denied, 875 P.2d 73 (Cal. 1994) (en bancX cert de- 
nied, 115S. Ct. 577 (1994). 
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phone numbers of all of the children who participated in the study; all drafts of 
the study design; all notes, memos, and videotapes pertaining to the study; the 
names, addresses, telephone numbers, background information, and occupa- 
tions of Eill interviewers; hard copy tabulations and data tapes; originals of all 
test materials; all correspondence relating to the research; the names, ad- 
dresses, and background information of all consultants; the names and ad- 
dresses of all funding sources; and the names and telephone numbers of all re- 
spondents who were excluded from the study. 

Given the published implications of my research, I had assumed that I 
might at some point be deposed about this study. I was, however, not prepared 
to receive a subpoena of this breadth and one that would require turning over 
the names of three- to six-year-old children. Such disclosure would have vio- 
lated written confidentiality agreements that I had signed with each parent be- 
fore conducting the research. 

I had also anticipated that the Medical College of Georgia (“MCG"), on 
whose faculty I was a full professor and under whose auspices the research had 
been conducted, would provide appropriate legal support for my position. 
However, Michael Bowers, the Attorney General of the State of Georgia and 
the official counsel for the medical school, took the position that the prevailing 
legal issue was not human subject confidentiality, academic freedom, or the 
reasonableness of the subpoena power, but rather the Georgia Opien Records 
Act, a law designed to permit public access to “official records.”" Mr. Bowers 
took this position even though RJR did not, at that time, request the records 
via the Open Records Act. I refused to comply with the subpoena and MCG 
refused to provide me with legal assistance. 

I contacted my own lawyer, Robert W. Hunter, III, who prepared a motion 
to quash the RJR subpoena.'* On April 28, 1992, Chief Superior Court Judge 
William M. Fleming, Jr., ruled in favor of our motion to quash." RJR immedi- 
ately appealed the ruling to the Georgia Court of Appeals, but that court, on 
February 9, 1993, ruled in our favor arguing that the requested documents were 
beyond the bounds of reasonable discovery.'’ 

Two weeks later, in an article in a local newspaper, MCG lawyer Clay 
Stedman stated that the school had not supported my legal efforts because of 
their position on the Open Records Act.'* Stedman said that MCG “decline[d] 
to object to [the] release of this information on the basis that although it was 
not an Open Records [Act] request. Open Records would have required us to 
release it.”'* Ironically, RJR attorneys did not know of MCG’s position on this 


14. Open Records Act, Ga. Code ANN. § 50-18-70 to -76 (Supp, 1996). 

15. Motion to Quash, Fischer v R.J. Reynolds Tobacco Co., No. 93-RCCV-230 (Ga. Super. Ct. 
Richmond County , Apr. 16, 1992). 

16. See Fischer v R.J. Reynolds Tobacco Co., No. 93-RCCV-230 (Ga. Super. Ct. Richmond 
County , Apr. 28, 1992) (order granting motion to qu^h). 

17. SeeR.J. Reynolds Tobacco Co. v Fischer, 207 Ga. App. 292 (1993). 

18. Kathleen Donahue, Researcher Has Hefty Legal Fees, THE AUGUSTA CHRONICLE, Feb, 27, 
1993, atI7A. 

19. Id 
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issue and had previously admitted in their Court of Appeals brief that they be- 
lieved the records were not accessible to them under the Open Records Act be- 
cause the research had not been supported by state funds.“ 

One week after the publication of this article, James R. Johnson, legal 
counsel for RJR sent a letter to H. Dean Propst, Chancellor of the University 
System of Georgia, and subsequently to Francis Tedesco, President of MCG, 
requesting that my research records be released to RJR under the Open Rec- 
ords Act.^' I was given forty-eight hours to turn over all of the previously de- 
scribed records with the exception of the children’s names. Clay Stedman, as 
MCG legal counsel, indicated that I would be suspended if I did not turn over 
the documents. Francis Tedesco, M.D., President of MCG, indicated that the 
Attorney General would have me arrested if I did not comply with the request. 

At the advice of my lawyer, I turned all of the documents over to the court 
for protection until such time as the legal issues relating to the Open Records 
Act, academic freedom, and human subject confidentiality could be resolved. 
The court accepted the documents and approved a temporary restraining order 
against the Open Records request.^ 

One month later, RJR petitioned the court to assist MCG and the Attorney 
General in the action against me.” Both the Attorney General’s Office and 
MCG supported RJR’s compelled disclosure motion.” Ironically, this action 
united the medical school and a tobacco company against one of the school’s 
own faculty members. 

On August 12, 1993, I received a nine-page letter listing documents and 
data requested by RJR through the Open Records Act.” It stated that RJR 
wanted all documentation related to the study regardless of when it was gener- 
ated or by whom.” In response to a 1993 change in the Open Records Act 
which excluded release of the names of research participants, RJR did request 
that the subject names be redacted from the submitted documents. 

On December 1, 1993, 1 resigned from the faculty of MCG and entered pri- 
vate practice in Augusta. On July 20, 1994, Judge John H. Ruffin signed an 
RJR request to release all of the records held by the court. The records were 
released to an RJR lawyer before we were notified of the decision, making an 
appeal of this decision moot. 


20. Petitioner’s Brief. R J. Reynolds Tobacco v. Fischer, 207 Ga. App. 292 (1993). 

21. Letter from James R. Johnson, Legal Counsel, RJR, to Francis J. Tedesco, President, MCG 
(Mar. 10, 1993) (on file with author). 

22. Motion for Temporary Restraining Order, Fischer, No. 93-RCCV-230 (Ga. Super. Ct. Rich- 
mond County , Mar, 12, 1993). 

23. 5ee James R. Johnson’s Motion to Intervene as a Defendant, Fischer, No. 93-RCCV-230 (Ga. 
Super. Ct. Richmond Counly, Apr. 22, 1993). 

24. Letter from David M. Monde, Attorney, Jones Day, ReavIs & Pogue, to Kathryn L. Allen, 
Senior Assistant Attorney General (Apr. 20, 1993) (on file with author) . 

25. Letter from RJR to author (Aug. 12, 1993) (on file with author). 

26. See id. 
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B. Lessons Learned 

Every day in every academic institution, people request information from 
scientists. Most of the time this is done by fellow scientists in the process of 
scientific research. For example, after the publication of the “Old Joe” study, I 
received requests from other researchers for specific information about our 
study and how it was done. Such requests are usually limited to information 
that would permit replication of the research. Successful replication is essential 
to establish scientific validity, and therefore scientists are usually pleased to 
share information. 

Scientists do not use subpoenas to seek scientific truth! Thus, the subpoena 
of a researcher’s files is evidence that the process has moved outside of the 
realm of scientific inquiry. As the cases cited in this paper illustrate, a sub- 
poena usually means that the research in question has commercial implications 
and that a company has decided that its lawyers, rather than its scientists, are in 
the best position to protect the company’s Interests. 

Nevertheless, many subpoenas for research are routine. For example, a 
medical researcher might discover and report a series of side-effects in patients 
taking a new drug. The pharmaceutical company that manufactures the drug 
may then subpoena the records to see if there is an alternative explanation for 
the patients’ symptoms. Other than concerns about patient confidentiality, 
such a subpoena would be handled in a routine fashion. 

However, not all compelled disclosure is routine. In the extreme, subpoe- 
nas can be unwittingly used in a manner that is damaging to the researcher, the 
scientific process, and the greater public good. 

Ill 

D.-VV1AGI.NG Effects of Extreme Subpoenas 

A. Discredit the Research. Discredit the Researcher. 

It was clear from the U.S.A. Today interview that RJR wanted to discredit 
me and my research." Furthermore, this refutation would not follow the usual 
“rules” of science. 

The standards for a published scientific paper require that the report in- 
clude sufficient detail about the scientific methods utilized so that another indi- 
vidual in the field could duplicate the study. This was precisely what Advertis- 
ing Age did after initially expressing reservations about the “Old Joe” research. 
They commissioned research that was published five months later and showed 
that the Camel campaign was indeed highly effective in reaching young people, 
especially children younger than age thirteen.^ The president of the research 
company said, “I was blown away by the number of smaller kids who could 


27. See RJ. Reynolds, supra note 6. 

28. 5eeGary Levin, Poll: Camel Ads Effective with Kids, ADVERTISING AGE, Apr, 27, 1992, at 12. 
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name cigarettes.”'*' Had RJR been concerned about the veracity of our find- 
ings, they could have duplicated our research in several weeks for a few thou- 
sand dollars. Instead, they spent two and a half years, and a great deal more 
money, in an attempt to access every page in my files. 

Why would RJR be interested in every scrap of paper in a research file? 
The answer to this question became clear from the experience of Dr. Joseph 
DiFranza, the lead author of one of the “Old Joe" studies.™ His research 
showed that Camel cigarettes’ share of the youth market increased from a mere 
0.5% to a substantial 32.8% following the “Old Joe" advertising campaign.'* 
Dr. DiFranza received a similar subpoena and turned over his records to RJR. 
In one of the letters to a colleague that was included in the disclosed docu- 
ments, Dr. DiFranza wrote, “I have an idea for a project that will give us a cou- 
ple of smoking guns to bring to the national media."'' RJR released this letter 
to the press and claimed that it proved that the researchers were biased and 
that the research was fraudulent." 

It is easy to characterize any scientist as being biased. The public assumes 
that scientists enter into research without a point of view. Nothing could be 
further from the truth. Science is impossible to do without passion about an 
idea. Scientists are not without opinions, but they agree to subject these opin- 
ions to objective experiments to see if they are true. In every researcher’s files, 
there are notes that could be taken out of context and characterized as proving 
bias. 

In addition, every research study represents a series of methodological deci- 
sions about how data are collected and analyzed. These decisions require ex- 
pert judgment and each of these judgments, when viewed in isolation, could be 
challenged. It is precisely because of this, that the final published paper be- 
comes the record of the research. In the published manuscript, the researcher 
must describe the findings, discuss their meaning, and most importantly, iden- 
tify the study’s limitations. 

The broad subpoena filed by RJR is akin to requiring a Supreme Court Jus- 
tice to report every private note made and every comment spoken in consider- 
ing a case, rather than merely being responsible for the contents of the final 
opinion. It would be quite easy to discredit the decisions of even the best 
judges if their private notes and thoughts were publicly opren on demand. 

B. Human Subject Confidentiality 

The conduct of research on human subjects requires that the public have 
confidence that its best interests will be protected and that its confidentiality 
will be preserved. In the case of our research, RJR requested the names and 


29. Id. 

30. DiFranza, supra note 1. 

31. See id. 

32. See Marla Mallory, That's One Angry Camel, BUS. Wk., Mar. 7, 1994, at 94. 

33. See id. 





225 


FISCHER. FMT 


08/13/97 2:45 PM 


Page 159; Summer 1996] SCIENCE AND SUBPOENAS 165 

addresses of 239 three- to six-year-old children whose parents had signed 
agreements in which we promised complete confidentiality. According to 
Peggy Carter, an RJR spokesperson, the company intended to use this informa- 
tion to contact the research subjects.^* Her reason for requesting this breach of 
confidentiality was that “ [tjhere have been a number of stories that have come 
up in recent years where scientists claimed to have produced research that . . . 
was never done at all.”’® While this reasoning is paranoid at best, it would not 
be necessary for RJR to knock on children’s doors at night to prove that the 
data in question were collected, rather than fabricated. 

The issue of subject confidentiality took an interesting legal turn in my case. 
MCG initially acknowledged tbe potential for abuse. In a letter from Carol 
Huston, one of tbe school’s attorneys, to the Attorney General’s office, she 
stated that 

[Fischer’s] concern, which I believe is well founded, is that Reynolds is attempting to 
harass him (and other researchers) through tactics such as this in order to discourage 
future research, the results of which may not be favorable to the tobacco industry. . . . 

We also believe if [RJR] obtains the names of the respondents, it seems very likely 
that [it] may contact them and attempt to harass them. This, in turn, may discourage 
other individuals from participating in future research projects. 

Despite these observations by an MCG lawyer, the Attorney General’s po- 
sition prevailed, and the school insisted that all names be released. 

As a general matter, institutions that participate in federally funded medical 
research must sign agreements with the Department of Health and Human 
Services (“DHHS”) , by which they agree to conduct research according to fed- 
erally-established guidelines. Human subject confidentiality is well-protected 
by these standards. My study, however, was not federally funded and was sub- 
ject to these guidelines only because of contractual agreements between DHHS 
and MCG. 

On September 8, 1992, I was contacted by the acting chief of the Office of 
Protection from Research Risks of the National Institutes of Health. He had 
heard of my case and wanted information about any breach of human subject 
protection. He subsequently sent a letter to the school alleging noncompliance 
with their DHHS contract because of the school’s position requiring release of 
my subjects’ names. The school responded that the federal regulations could 
be avoided because my research Wcis not federally funded. DHHS and MCG 
subsequently signed a revised contract in which only federally funded research 
was governed by federal regulations regarding subject confidentiality. 

C. Harassment 

The tobacco industry approach to litigation has been described by Lawton 
M. Chiles, Jr., Governor of the State of Florida, as “designed to confuse the 
medical evidence, stone-wall, delay, refuse reasonably to settle claims, and to 


34. See Marcia Barinaga, Who Controls a Researcher’s Files? Tobacco Company R.J. Reynolds 
Subpoenas Research Study Data, 1256 SCL 1620 (1992). 

35. Id 
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run up plaintiffs’ attorneys’ fees in a war of attrition. He cites a memo writ- 
ten by J. Michael Jordan, an attorney for RJR: 

The aggressive posture we have taken regarding depositions and discovery in general 
continues to make these cases extremely burdensome and expensive for plaintiffs’ 
lawyers, particularly sole practitioners. To paraphrase General Patton, the way we 
won these cases was not by spending all of Reynolds’ money, but by making the other 
son of a bitch spend all his." 

This same approach was used to wear down my resources, including my 
time, attention, and money. The ultimate goal is to make the process suffi- 
ciently painful so that the researcher cannot complete further research and so 
that other scientists are discouraged from conducting similar studies. 

Scientists are perfect subjects for harassment by litigation. They often have 
little knowledge of the law and little patience for the slow and subtle workings 
of the legal system. The distraction and anxiety caused by depositions, legal 
costs, and court appearances can eaisily put an abrupt end to a promising line of 
research or a research career. 

It should be noted that RJR did not limit its harassment efforts to the use of 
the press and the courts. It also attempted to conscript the institution at which 
I worked. Bernard Wagner, M.D., Professor at the New York University 
School of Medicine and paid consultant to RJR, contacted my research col- 
leagues and the President of MCG with accusations of scientific fraud.“ A 
similar letter was sent to the University of Massachusetts regarding Dr. Di- 
Franza’s “Old Joe” study.’’ While MCG did not respond, the University of 
Massachusetts used these baseless accusations to initiate scientific misconduct 
hearings against Dr. DiFranza. He was eventually found innocent of these 
charges." 


IV 

Suggestions 

.4s a researcher who has been through the experience of compelled disclo- 
sure. many of the suggestions outlined in this paper do not appear to be viable 
solutions to the problem that I faced. I would not argue that scientists deserve 
specitil protection under the law in the same way that lawyers, priests, or jour- 
nalists have claimed the need for protection of their relationships with clients, 
parishioners, and confidential sources. Science, after all, is based on a shared 
and open search for truth. I am not, however, so naive as to believe that most 
subpoenas for research records are based on goodwill, public interests, or the 
search for truth. I offer the following thoughts: 


36. Complaint, Florida v. American Tobacco Co. et al.. No. CL-1466A0 (Circuit Ct.,15th Circuit, 
Palm Beach. Fla., Apr, 18, 1995) 

37. Id at 28-29 (memorandum from J. Michael Jordan, legal counsel. RJR). 

38. Letter from Bernard Wagner to Tina Rojar (Mar. 29. 1993) (on file with author). 

39. Based on the author's conversations with Dr. DiFranza. 

40. Id 
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First, if a request for compelled disclosure has been made, realize that the 
process has moved outside of the normal exchange between scientists. It is 
likely that a commercial entity and its profits are at stake. It is also likely that 
the company will have greater legal resources and experience than the scientist, 
who may have never stepped foot inside a courtroom. 

Second, despite institutional affiliation and responsibilities to protect aca- 
demic freedom, universities may provide poor legal counseling to scientists 
facing compelled disclosure. This problem may become greater due to the in- 
creased reliance of universities on corporate support. We might expect to see 
university presidents siding with corporate contributors rather than their aca- 
demic faculty. 

Next, if a subpoena is requested by an industry, consider the industry’s past 
record in dealing with the scientific community. Consider whether the industry 
has used the legal system to discourage good science in the past. 

Also, consider the breadth of the request. If it goes far beyond what a rea- 
sonable scientist would require to duplicate the research, then there may be 
other ways that the company could validate the research findings without vio- 
lating the privacy of the scientist’s records. 

Ask the scientist to identify specifically how compelled discovery could im- 
pede his research. It is impossible for the court to balance the rights of the 
company with those of the scientist unless it understands the implications of the 
legal process on the scientist’s time, attention, and financial resources. 

Finally, human subject confidentiality, promised as part of the research 
process, must be protected at all costs. There are excellent ways to identify sci- 
entific fraud without violating anonymity, such as the use of an independent re- 
view panel of scientists. 


V 

Conclusion 

The uneasy relationship between law and science is likely to continue re- 
garding disclosure of scientific research materials. Law and science are worlds 
apart in terms of values that they hold and the rules that they follow. Whether 
it be DNA evidence or silicone breast implants, it appears that these two 
worlds will collide with ever-increasing frequency. This inevitable collision will 
require that scientists have a better understanding of the legal implications of 
their research and that Judges have a better understanding of the impact of 
their decisions on the progress of science. 
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Dear Chairman Smith, Ranking Member Johnson, Chairman Sehweikert and Ranking Member 
Bonamici, 

On behalf of the Natural Resources Defense Council, I am writing to provide information 
that I hope will inform the Environment Subcommittee members’ consideration of topics for 
your February 1 Itb bearing on the discussion draft of a bill entitled the “Secret Science Reform 
Act of 2014,” 

The discussion draft of the bill is deeply troubling and should be rejected by 
subcommittee members. Tbe draft legislation would effectively amend numerous environmental 
statutes, and it marks a radical departure from longstanding practices. Its end result would be to 
make it much more difficult to protect the public by forcing EPA to ignore key scientific studies, 
including those submitted by industry. 

The bill proceeds from a faulty premise from which it then undermines EPA’s ability to 
carry out its most basic responsibilities. The notion of “secret science” is a canard and ignores 
longstanding practices, recognized in law, that protect patient information, intellectual property 
and industrial secrets. This letter inventories some of the key ways such information is used, and 
needs to be used by EPA. The Subcommittee has done nothing to demonstrate how the public 
has suffered as a result before seeking to overthrow law and practice. But it easy to show how 
the public would suffer if the bill’s proscriptions and restrictions were put into effect. 

This letter will elaborate on these points: 

• The whole notion of "secret science,” based on studies of fine soot pollution conducted 
almost two decades ago, is unfounded. 

• The bill would make it impossible for EPA to use many kinds of studies that it 
necessarily relies on to protect the public because those studies use data that has long 
been understood to be legitimately confidential. 

• The bill would make it impossible for EPA to use many kinds of economic models it 
routinely relies on because those models are proprietary. 

• The bill advantages industry by exempting from its coverage EPA activities where 
industry is the primary party likely to submit confidential information, such as permitting. 

Nonetheless, the bill would make it harder for EPA to consider confidential information ftom 
industry in many instances, limiting the agency’s ability both to protect the public and to reduce 
the costs of regulation. 

Covered Actions 


The draft bill defines a “covered action” to mean “a risk, exposure, or hazard assessment, 
criteria document, standard, limitation, regulation, regulatory impact analysis, or guidance.” This 
definition creates a fundamental double standard biased in favor of corporations and against 
public health and safety. The draft legislation (I) restricts the information EPA can use to take a 
series of actions to protect public health and the environment, while it (2) simultaneously leaves 
untouched a host of actions that industry needs and desires — notwithstanding that these industry- 
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favored actions often rely on industry-supplied scientific and technical information that industry 
may shield from the public. 

Consider just a few examples of EPA actions that industry wants or needs EPA to take, 
and that do not fall under the definition of “covered action.” For these actions, EPA can 
continue to rely on so-called “secret science” supplied by industry that remains shielded: 

• Industry permit approvals, revisions and renewals under the Clean Air Act, Clean Water 
Act and RCRA; 

• Industry pesticide registrations, exemptions, and tolerances under the Federal Insecticide, 
Fungicide, and Rodenticide Act (FIFRA); 

• Applicability determinations under EPA statutes and adjudications under the 
Administrative Procedure Act that determine whether regulations do or do not apply; 

• Requests under some EPA regulations for industry exemptions that may be granted 
without need for proposed or final regulations by the agency; 

• Certifications and compliance reports for vehicles, engines and equipment for various 
Clean Air Act motor vehicle regulations. 

The draft legislation exempts all of these industry-desired or needed agency actions from the 
bill’s strictures as well as from the bill’s purported concern for transparency. 

Examples of Health Protection s That the Draft Bill Would Obstruct 

The following examples are drawn from just some of the statutory responsibilities and 
authorities that EPA has under current law. The draft bill would limit EPA’s ability to review 
relevant information that current law allows EPA to consider to protect public health, safety and 
the environment: 

• EPA could not establish a drinking water standard or health advisory for a contaminant 
under the Safe Drinking Water Act based on information that industry claims was 
protected confidential business information (CBl). 

• EPA could be hindered in responding to emergency situations. For example, initially 
some of the data on the chemical Freedom Industries spilled last month in West Virginia 
was not publicly disclosed. It was eventually released in response to a letter from 
Congressman Waxman to the manufacturer of the chemical, Eastman Chemical. The 
draft legislation is problematic in the extreme by allowing industry to decide selectively 
what information EPA can use to issue a health advisory or a risk or hazard assessment, 
based on industry claiming that information to be CBl. 

• EPA could not establish a drinking water standard or health advisory based on 
epidemiological evidence or clinical studies where the medical records of the patients 
were confidential under the Health Insurance Portability and Accountability Act 
(HIPAA) or other patient confidentiality requirements, or where the study would not be 
“reproducible” because of restrictions on access to confidential patient information. 
These confidentiality safeguards for patient data are routine in the field of medical 
research, yet the draft legislation renders important advances and understandings in 
health and environmental research off-limits to EPA when carrying out the law to protect 
Americans. 
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• EPA could not issue a risk/hazard assessment or a cancellation of a pesticide based upon 
(1) studies containing CBI; (2) epidemiological or clinical studies where the medical 
records of the patients are confidential under HIPAA or other patient confidentiality 
requirements; or (3) where the study would not be “reproducible” because of restrictions 
on access to confidential patient information. For example, studies completed by 
Columbia University doctors have shown certain pesticides used indoors harm pregnant 
mothers and their fetuses, causing smaller head circumferences, and interfering with 
children’s brains’ development as they grow up. These patient records have been 
aggregated and published in peer-reviewed journal literature, but underlying medical 
records are required to be kept confidential under HIPAA and agreements with patients. 

• EPA could not regulate or issue guidance to prevent lead poisoning of children in housing 
being renovated, or lead-contaminated water or plumbing, based upon clinical and 
epidemiological studies, where the medical records of the patients are confidential under 
HIPAA or other patient confidentiality requirements, or where the study would not be 
“reproducible” because of restrictions on access to confidential patient information. For 
example, many of the studies of the adverse impacts of lead follow patients who have 
been exposed to lead, and those records would be protected from public disclosure. 

• EPA could not conduct risk/bazard assessments necessary to inform and govern tbe 
cleanup of Superfund sites, to the extent that potentially responsible parties asserted CBI 
protections over company information potentially implicating their contribution to a site, 
or CBI relating to specific chemicals. The draft legislation thus would allow any 
assertion of confidentiality claims by responsible parties engaged in Superfund cleanups 
to delay or thwart those cleanups in local communities, including the jobs associated with 
those activities. 

In each of these examples, the draft legislation would mark a radical retreat from current law, by 
preventing EPA from considering key studies in deciding how to protect public health, safety 
and the environment. 

Hazard Assessments and Imminent and Substantial Endaneerment 

The draft bill would prohibit EPA from taking actions under federal laws like the 
Resource Conservation and Recovery Act (RCRA) and the Clean Air Act to protect Americans 
against “imminent and substantial endangerment,” to the extent EPA relies upon any health 
studies involving confidential patient data or relies upon industry CBI. The latter could include 
industrial chemical or product formulations, process data, industry testing or research or trade 
secrets. EPA must conduct hazard and risk assessments to understand the nature of chemical and 
oil spills, explosions or other hazards endangering the public. Under current law, there are no 
restrictions on EPA conducting those hazard assessments, protecting the industry CBI and 
safeguarding the public. The draft legislation radically changes that. To the extent that any 
information covered by the draft bill is relied upon by EPA, the agency could not act against 
imminent and substantial endangerment of public health nor could EPA even "disseminate" 
warnings to the public. 
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“Dissemination.” Censorship and Reckless Retroactivity 

The draft bill’s astonishingly broad language prohibits EPA from “disseminating” any 
“risk, exposure, or hazard assessment, criteria document, standard, limitation, regulation, 
regulatory impact analysis, or guidance” that relied on scientific and technical information 
meeting the bill’s criteria. This language produces the perverse result that EPA would be barred 
from publishing on its website — or indeed even in the Code of Federal Regulations — prior and 
existing regulations, reports, guidance, risk, exposure or hazard assessments that relied on 
scientific and technical information before the draft bill’s consideration. This results in a 
reckless retroactivity and censorship of duly enacted regulations and agency reports that one 
cannot imagine even the draft legislation’s authors intended. (Of course, prohibiting EPA from 
disseminating adopted regulations would not cause those regulations to be repealed; it would just 
make it immeasurably harder for anyone to find and follow the law.) But that is the consequence 
of the plain language of the draft bill, and such a “dissemination” prohibition would result in the 
massive censorship of valuable public health and safety information. 

Illegal Delay and the Circular Problem of “Reproducibility” 

The draft bill prohibits EPA from taking any covered actions unless all scientific and 
technical information relied on is “publicly available in a manner that is sufficient for 
independent analysis and substantial reproduction of research results.” The perverse problem 
with this language is that it could be read to mean that the only way to know with any certainty 
whether information is sufficiently reproducible is to allow time for independent parties to 
attempt to reproduce those research results. We know from experience that this can take years 
and involve great expenses. 

The draft bill’s prohibition thus would prevent EPA from complying with statutory 
deadlines created by Congress under numerous federal laws. Before EPA may even propose or 
finalize a regulation to meet a statutory deadline, the agency would need to await confirmation of 
reproducibility, or else face constant anti-regulatory attacks from the earliest stages of a 
rulemaking that some scientific or technical information is not reproducible. This dynamic 
would poison EPA ralemakings cither with massive delay or inescapable uncertainty, 
fundamentally obstructing EPA’s responsibilities under its various statutes to protect human 
health and the environment. 

Moreover, this provision could actually create a perverse incentive for regulated 
industries with the financial means to do so cither to (1) not undertake efforts to reproduce 
research results, so they may continue to charge that results are not reproducible; or (2) withhold 
from EPA research results that do prove the information is reproducible. And of course 
members of the public that lack the resources to conduct such reproduction studies, that want 
EPA to protect public health and the environment, will be unable to clear this hurdle in the draft 
bill. 

Regulations Granting Industry Flexibility or Regulatory Relief 

Industry sometimes appeals to EPA during the course of proposed miemakings, or even 
prior to the initiation of rulemaking, to loosen the rigor of agency regulations, accord industry 
operational flexibilities, extend compliance deadlines or take other actions to reduce alleged 
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regulatory burdens. Frequently industry does so by submitting information particular to a 
specific company or industry sector; a particular chemical or product formulation; or a particular 
process unit or manufacturing process. These submissions frequently are accompanied by claims 
that information is CBI, due to the company-specific or industry-specific nature of information 
that may be proprietary, confidential or trade secrets. Industry parties sometimes submit health 
studies or risk assessments they have conducted that may contain confidential clinical data or 
other information that they do not wish to make publicly available. 

The draft legislation would create a dynamic in which EPA is unable to consider that CBI 
or otherwise confidential health or risk data in deciding whether to adopt regulations or issue 
guidance that grants industry the requested regulatory flexibilities. When EPA exercises its 
regulatory authorities, at least, the draft bill also constrains the agency’s ability to be flexible or 
relieve regulatory obligations, precisely where it might be needed most; by being responsive to 
particular demonstrations made by specific companies based on persuasive information that also 
happens to be CBI. It does not appear that the draft bill’s co-sponsors could have intended this 
outcome. 

Proprietary Models 

The bill prohibits EPA from taking covered actions to enforce the law and protect the 
public if doing so involves relying on “computer codes and models” for creating and analyzing 
scientific and technical information. Section 6(h)(3)(B). This provision has the perverse effect 
of barring EPA from relying on proprietary nurdels or computer programs whose software, 
design features and other inputs were created by and are owned by the private sector. There are 
undoubtedly numerous proprietary models used by EPA,' but a widely used model under the 
Clean Air Act serves as a useful example to highlight the bill’s irresponsible — and probably 
unintended — consequences. 

The Integrated Planning Model (1PM) is the most widely used model “to analyze the 
impact of air emissions policies on the U.S. electric power sector.”^ It is employed by EPA, state 
governments, the private sector and public interest organizations, and was developed by ICF 
Consulting, Inc., which owns the rights to the model and its utilization. EPA explains the 
purpose of the IPM and its value thusly: 

EPA uses the Integrated Planning Model (IPM) to analyze the projected impact of 

environmental policies on the electric power sector in the 48 contiguous states and the 

District of Columbia. Developed by ICF Consulting, Inc. and used to support public and 


' For other examples of proprietary models employed by EPA, see 

http://www.epa.gov/pesticides/science/models pg.htm . The agency has said that “EPA prefers 
using non-proprietary models when available. However, the Agency acknowledges there will be 
times when the use of proprietary models provides the most reliable and best-accepted 
characterization of a system.” htt p://www.epa.gov/crem/librarv/cred guidance Q3Q9.Ddf . at 3 1 . 
We respectfully submil that EPA should be asked to identify all proprietary models used by the 
agency, and how restrictions on their use would impede the agency’s ability to enforce the law 
and protect public health and the environment. 


http;//www.epa.gov/powersectormodeling/. 
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private sector clients, IPM is a multi-regional, dynamic, detenninistic linear 
programming model of the U.S. electric power sector. It provides forecasts of least-cost 
capacity expansion, electricity dispatch, and emission control strategies for meeting 
energy demand and environmental, transmission, dispatch, and reliability constraints. 

IPM can be used to evaluate the cost <md emissions impacts of proposed policies to limit 
emissions of sulfur dioxide (SO 2 ), nitrogen oxides (NO^), carbon dioxide (CO 2 ), 
hydrogen chloride (HCl), and mercury (Hg) from the electric power sector. 

The IPM relies on computer codes and model characteristics whose content, features, inputs and 
other elements are not “specifically identified” and “publicly available in a manner that is 
sufficient for independent analysis and substantial reproduction of research results.” 

Thus, the draft bill would prohibit EPA from proposing, finalizing or disseminating 
covered actions if it relied on the IPM, or it would require EPA to abandon use of the IPM 
altogether. This would produce the following harmful outcomes: 

• When proposing or finalizing regulations, regulatory impact analyses or other covered 
actions, the draft bill would prohibit EPA from using the sophisticated IPM to analyze the 
projected impact of its power plant regulations on the electricity grid and its reliability, 
transmission lines, dispatch. Jobs in the power and coal mining sectors, emissions control 
and retirement decisions, among other information generated by the 1PM; 

• The draft bill would prohibit EPA from “disseminating” to Congress, the public, industry 
officials and state and local government any covered action (such as a regulatory impact 
analysis) that contained or relied upon any information generated from the proprietary 
IPM; 

• The draft bill would prohibit EPA from proposing or finalizing regulations to lessen 
regulatory impacts on the power sector, adopt exemptions or issue flexibility guidance to 
the extent that EPA relied upon the proprietary IPM; 

• The draft bill would prohibit EPA from conducting risk, exposure or hazard assessments 
at the request of Congress to analyze the impact of proposed Clean Air Act legislation or 
EPA regulations on the power sector, or “disseminating” such results to Congress, to the 
extent that EPA relied on the IPM; 

• Had the draft bill been enacted into law at the time, the Bush administration would have 
been unable to supply members of Congress or the public with all the useful IPM results 
generated to assess the impacts of Clear Skies legislation in the House and the Senate, as 
well as the Bush administration’s Clean Air Interstate Rule and Clean Air Mercury Rule.^ 
Indeed, members of Congress, President Bush and administration officials drew heavily 
upon these IPM results in promoting the Clear Skies bills during congressional 
deliberations and in statements from their offices."* 


^ Information still available on EPA’s website demonstrates the vast extent to which the Bush 
administration relied upon the IPM to analyze the Clear Skies bills as well as EPA’s related 
regulatory actions. See, e.g.. http://www.eDa.gov/clcarskies/tech adden.ndf : 
htt p://www.epa.aov/clearskies/tech addendum.pdf ; & 
http://www.epa.BOv/clearskies/clearskiessummarv04-l 1. pdf . 

* See, e.g., 

http://vosemite.epa.aov/opa/admpress.nsfy6427a6b7538955c585257359003fD230/clbl 1 Ib0d87d 
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Another example of an EPA model that the draft legislation likely would render 
unavailable is the agency’s use of various physiologieally based pharmacokinetic (PBPK) 
models to conduct chemical assessments under the Integrated Risk Information System (IRIS). 
EPA says that “these models represent an important class of dosimetry models that are useful for 
predicting internal dose at target organs for risk assessment applications,”’ It is likely that some 
widely-employed PBPK models would not pass muster under this draft legislation, due to their 
proprietary nature, the public unavailability of information or the inability to sufficiently 
reproduce model results. 

In one recent example, EPA relied upon a PBPK model to propose non-cancer risk 
estimates for methanol at, or nearly at, an order of magnitude weaker than those proposed 
previously. The draft legislation could prohibit EPA from relying upon this PBPK model to 
lower the risk estimates for methanol. Moreover, any other attempt by industry to persuade EPA 
to weaken risk assessments for chemicals in IRIS could not rely upon PBPK models failing to 
meet the draft bill’s criteria. Nor could those industry efforts rely upon health studies, risk 
assessments, research, product or process information or business information claimed by 
industry to be confidential. The draft bill would make this true for all risk, hazard and exposure 
assessments under IRIS and other EPA programs. 

Finally, the draft bill is so poorly drafted that it could conceivably prevent EPA from 
using commercially available software to carry out basic computing functions, because the 
computer codes behind that software are proprietary and not publicly available. Again, we do 
not believe this absurd result was intended by the authors of the draft legislation, but this is the 
plain reading of its language. 

Obstructing Clean Air Act Enforeement 

The draft legislation, coupled with the unwarranted subpoena steps by the Committee 
majority, plainly is targeting a few clean air health studies that show causal associations between 
fine soot pollution (PM 2 , 5 ) and premature mortality. One of the draft bill’s co-sponsors has 
suggested that the massive body of scientific evidence showing a causal association between soot 
pollution and mortality comes down to “secret” data from just two studies.’ This is inconect. A 
much broader body of scientific studies examines and reaffirms the causal association between 
fine soot pollution and mortality. These studies post-date the so-called "Harvard Six Cities" and 


591385256c0500625054!ODcnDocument : http://www.inhofe.senate.gov/epw- 
archive/press/bchairman-inhofe-introduces-the-administrations-clear-skies-initiative/b : & 
http://www.epw.senate.gov/107th/smi 061202.htm : http://georgewbush- 
whitchouse.archives.eov/news/releases/2002/02/200202 1 4-5.html . 

’ http://cfpub.epa.gov/ncea/cfm/recordisttlav.clm?deid'=l 35427 . 

’ Rep. Lama Smith, “The EPA’s Game of Secret Science,” The Wall Street Journal (July 29, 
2013). 

htt p://online.wsi. com/news/articles/SB 1 00014241278873238291 0457862456200823 1682?mg=r 
eno64- 

wsi&url=http%3A%2F%2Fonline.wsi.eom%2Farticle%2FSB 10001 424127887323829 10457862 
4562008231682.html . 
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"American Cancer Society" studies, some of them independently re-analyze the studies, and they 
consistently find the same causal soot-mortality relationship/ 

Committee Chairman Smith has charged that the data in the Harvard and American 
Cancer Society studies “have not been subjeeted to scrutiny and analysis by independent 
scientists.”* This too is incorrect. 

In December 2012, a seminal report entitled the 2010 Global Burden of Disease^ 
"estimate[d] over 2. 1 million premature deaths and 52 million years of healthy life lost in 20 1 0 
due to ambient fine particle air pollution, fully 2/3 of the burden worldwide," Drawing upon a 
broad body of data and studies from around the world, the report examined the risks of 
premature mortality linked to soot pollution and independently affirmed the results of the 
Harvard Six Cities study. The Global Burden of Disease researchers found significant mortality 
impacts from fine particulate pollution. They concluded that “[t]he magnitude of disease burden 
from particulate matter is substantially higher than estimated in previous comparative risk 
assessment analyses.” 

As explained in a release"’ by the esteemed Health Effects Institute, a contributor to the 
report, “[t]he 2010 [Global Burden of Disease report] was produced by a rigorous scientific 
process involving over 450 global experts and led by the Institute of Health Metrics and 
Evaluation (IHME) at the University of Washington along with its partner institutions: the World 
Health Organization, the University of Queensland, Australia, Johns Hopkins University, and 
Harvard University.” 

Similarly, in July 2000. the Health Effects Institute issued a special report" entitled 
“Reanalysis of the Harvard Six Cities Study and the American Cancer Society Study of 
Particulate Air Pollution and Mortality.” The explicit goal of that study was “to conduct a 
rigorous and independent assessment of the findings of the Six Cities and ACS Studies of air 
pollution and mortality.” (p.ii) To accomplish this goal, the team of researchers had “access to 
the original data” once they entered into contractual agreements and a Memorandum of 
Understanding to ensure that confidentiality was protected, (p.4). The report concluded that 


^ In revising and updating National Ambient Air Quality Standards (NAAQS) for fine particulate 
matter, EPA devotes an entire chapter of its Regulatory Impact Analysis (RIA) to cataloguing 
and reviewing updated health effects studies, and explaining how they were incorporated into the 
agency's 2012 standards review. See, e.g., http://www.epa.gov/ttn/ecas/regdata/RIAs/fmalria.pdf 
(at pp. 5-7 to 5-8 listing 5 updates from the proposed 2012 RIA; fig 5-4 at p. 5-73; pp. 5-31 to 5- 
35). 

* Supra note 6. 

^ http://www.thelaneet.eom/ioumals/lancet/article/PllS01 40-6736( 1 2J6 1 766-8/lulltext . 

http://www.healtheffects.ore/Intemational/GBD-Press-Rclease.pdf . The Health Effects 
Institute is "a nonprofit corporation chartered in 1980 as an independent research organization to 
provide high-quality, impartial, and relevant science on the health effects of air pollution." 
Funded jointly by the federal government and industry, it is an honest broker that has garnered 
widespread respect for its scientific expertise, integrity and research excellence. 

" htt p://pubs.healtheffeets.org/getfile.php?u=274 . 
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“reanalyses assured the quality of the original data, replicated the original results, and tested 
those results against alternative risk models and analytic approaches.” (pp.iii-iv). 

EPA's Integrated Science Assessment'^ for the PMj s standards explained (p. 7-95) that 
the Harvard and ACS studies have “undergone extensive independent reanalysis,” and “were 
based on cohorts that were broadly representative of the U.S. population.” Reviewing this 
assessment and the broader body of epidemiological and toxicological studies, EPA's official 
Clean Air Science Advisory Committee (CASAC) recommended “‘upgrading’ the causal 
classification for PM 2.5 and total mortality to ‘causal’ for both the short-term and long-term time 
frames.” CASAC further found “[t]here are epidemiological studies showing a positive 
association of all-cause mortality with PM 2 . 5 .” 

Despite this extensive body of evidence, thorough re-analysis, and reaffirmation by 
governmental scientific advisory bodies, the draft bill is founded on an obvious agenda to deny 
EPA the ability to rely upon peer-reviewed medical studies that involve commitments to patient 
confidentiality, when the agency carries out its statutory responsibilities to safeguard public 
health and clean air. The tmth is there is a basic difference between “secret science” and 
confidential patient data subject to confidentiality agreements reached to conduct important 
medical research. The American people understand this difference. The legitimafe researchers 
and reanalysis initiatives that committed to the confidentiality policies of the relevant research 
institutions, as HEI and the Global Burden of Disease teams did, were able to access the patient 
data. 


EPA has squarely rejected the effort to create doubt through secrcey charges concerning 
these same health studies: 

The EPA is transparent with regard to the scientific bases of agency decision making and 
disagrees with assessments and your assertion that the agency relies on ‘secret’ data in 
regulatory actions and of health benefits. In setting the National Ambient Air Quality 
Standards (NAAQS) and in assessing health benefits anticipated from air pollution 
regulations, the EPA relies on the scientific studies that arc published in the peer- 
reviewed literature. The EPA provides the information used in regulatory decisions, 
including the epidemiological studies, in the publicly available docket accompanying 
each rulemaking." 

The Committee has now gone so far as to use its unfounded charges to write a bill that would 
block the use of a breathtaking range of science that has long been used to safeguard the public. 

Technology-Based Emission Standards 

The draft legislation would thwart EPA’s responsibility to carry out health safeguards 
required by Congress under the Clean Air Act and Clean Water Act. For example, both of these 
statutes contain “technology-based” emission standards for industry based on emissions 


" http://effaub.epa.gov/ncea/CFM/recordisDlav.cfm?deid=216546#Download . 

http://switchboard.m'dc.org/blogs/iwalke/04-10- 

13%20EPA%201etter%20to%20Scnator%20Vitter.pdf . 
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reductions deemed achievable by state-of-the-art technology.’'' EPA sometimes solicits from 
corporations information about an industrial sector’s pollution control technology, process units 
and other types of regulated or potentially regulated equipment. Industry requests that some of 
the information it submits to EPA be treated as CBI. Similarly, when industry representatives 
submit comments in response to proposed technology-based emissions standards, these 
commenters request that various information contained in those comments be treated as CBI. 

The draft bill would create a perverse dynamic in which corporate officials could thwart 
EPA’s development of statutorily required technology standards, by designating as CBI 
information that is crucial to determining what emissions rednetions are achievable by state-of- 
the-art technology. Indeed, the draft bill’s design would particularly obstmet the implementation 
and enforcement of technology-based safeguards for air and water, because industry 
representatives could so easily seek to designate a wide variety of technology and process 
information to be CBI. Accordingly, even though the draft bill does not purport to amend the 
Clean Air Act or Clean Water Act, and even though your Committee lacks the jurisdiction to do 
so, the draft bill would have the effect of radically re-working and weakening the purpose and 
effectiveness of these laws. 

Toxic Substances Control Act 

The draft bill would fundamentally obstruct EPA’s responsibility to protect the public by 
regulating toxic substances under the Toxic Substances Control Act (TSCA), which relies 
extensively upon industry claims of confidential business information. 

For example. Section 8(e) of TSCA requires chemical manufacturers, importers and 
processors to report immediately to EPA whenever they obtain evidence “that reasonably 
supports the conclusion that [a substance or mixture] presents a substantial risk of injury to 
health or the environment.” 

Typically, these industry reports claim the information provided is protected confidential 
business information — including the identity of the chemical, the name of the company 
submitting the information, as well as health and safety studies about the chemical.'^ The most 
recent list of section 8(e) studies from April 2013 shows just how pervasive these industry CBI 
claims are.'® 

Members of the public can only see the sanitized version of the 8(e) reports, which might 
show the results of lab testing for human or aquatic toxicity and which “reasonably support the 


''' See, e.g.. Clean Air Aet section 1 12(d) (Maximum Achievable Control Technology (MACT) 
standards). 

EPA has allowed these CBI claims to be asserted even though TSCA section 14(b) does not 
allow it. The current abuse of CBI under TSCA is a widely recognized problem. EPA is not 
required even to review all CBI submissions for their validity. There is no up-front justification 
requirement that must accompany CBI claims. Once CBI status is granted under TSCA it has no 
sunset and is rarely if ever re-opened. This has resulted in massive overuse and abuse of the CBI 
designation. For more information, see, e.g., http://blogs.edf.org/health/2010/02/12/worse-than- 
we-thought-decades-of-out-of-eontrol-ebi-claims-under-tsca/. 

http://www.epa. gov/opDt/tsca8e/pubs/8cmonlhlvretX)rts/2013/8eapr2013.html. 
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conclusion that [the substance] presents a substantial risk of injury to health or the environment.” 
(emphasis added). Although the public will not have access to this information, EPA will, and 
they use 8(e) reports to prioritize chemicals for greater reporting, or testing, potential regulation, 
potential voluntary agreements with companies to restrict or phase out the use of particular 
substances, as well as possible enforcement actions. 

A very similar function occurs under the new ehemicals program of TSCA (Section 5). 
Industry officials submit Pre-Manufacturing Notices and claim that information about their 
proposed new chemicals is CBI. This includes health and safety studies that should not be 
eligible for treatment as CBI under TSCA, but that EPA routinely treats as CBI anyway. While 
the public does not see information submitted as CBI, the agency does, and can use that 
information to take several steps: (1) reject a PMN, for example if the new substance is 
persistent, bioaccumulative and toxic; (2) require additional testing under a TSCA section 5(e) 
consent order; or (3) restrict some uses of the new chemical using a Significant New Use Rule 
(SNUR). 

The draft legislation irresponsibly prohibits EPA from taking or even proposing to take 
the aforementioned actions by relying on the submitted industry information to the extent that 
industry claims it to be CBI. This creates the perverse result that industry is allowed to prevent 
EPA from taking necessaiy steps to address “substantial risk of injury to health or the 
environment” caused or potentially caused by the industry’s own chemicals, based on the 
decision entirely within industry’s control to designate submitted information as CBI. And the 
particular perversity of the draft legislation is that information may well be CBI under current 
law; but current law does not restrict EPA from protecting the public simply because industry 
has legally protected interests over its CBI. 

Consider the following example under TSCA. A chemical manufacturer submits a Pre- 
Manufacturing Notice for a new chemical under TSCA Section 5, and the notice contains data or 
information that the manufacturer claims to be CBI. 

EPA has 90 days (plus an option for a 90-day extension) to review the notice and 
determine whether or not it wants to allow the new chemical to start being manufactured, 
whether it wants to require more testing, impose some restrictions, or stop the chemical entirely. 
If EPA takes no action on a PMN within the 90-day review period, the company submitting the 
notice can begin to manufacture the chemical. Once a new chemical is allowed to be 
manufactured, the chemical is then added to the TSCA inventory. This allows any other 
company to begin using the chemical for any other purpose (including in greater volumes than 
proposed in the original notice, and for different kinds of uses, including uses that may be much 
more dispersive and lead to greater human exposure, e.g., in a flame retardant). 

The definition of “covered action” in the draft legislation does not include inaction by 
EPA. Accordingly, the chemical manufacturer and other industrial users that follow-on may 
begin manufacturing new chemicals based upon the submission of CBI — “secret science” to use 
the nomenclature of the draft bill — all without any of that information needing to be publicly 
available or reproducible when EPA fails to take any action on receipt of the notice. 

If EPA does have health and safety concerns, however, based in part on the information 
submitted as CBI, TSCA authorizes EPA to take several steps: (1) require the company to do 
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more testing; (2) impose restrictions on the original notice submitter; and (3) restrict other 
entities from using the chemical for different uses or different volumes. 

The draft legislation treats all of these EPA actions under TSCA as “covered actions,” 
because they involved proposed or final regulations and/or the need for risk or hazard 
assessments. Accordingly, the draft bill prohibits EPA from taking any of these actions to 
protect the public, to the extent the agency needs to rely upon the industry CBI that raised the 
concerns in the first instance. 

So the draft legislation is an irresponsible one-way ratchet: industry may proceed to 
manufacture new chemicals based on EPA’s consideration (or even non-consideration) of 
“secret” CBI. But EPA may not regulate identified dangers or risks to the public from those 
chemicals based on consideration of that same “secret” industry CBI. 

Conclusion 


In sum, this draft legislation would effectively amend numerous environmental statutes in 
a manner that would obstruct the development and implementation of health and environmental 
safeguards. It would do so in a fashion that would also restrict industry’s ability to inform EPA 
decision-making, potentially raising the costs of regulation. At the same time, the bill unfairly 
caters to industry by exempting permitting and other agency actions from its ambit and 
underscoring the CBI protections in existing law. 

The Subcommittee ought to abandon this misguided project of chasing the phantom 
notion of “secret science.” With this draft bill, the Subcommittee has moved from reviving 
baseless charges about clean air science that were disproved over a decade ago to damaging 
EPA’s ability to use science for decades ahead. Surely there are more productive ways to spend 
its time. 


Sincerely, 


John Walke 

Clean Air Director 

Natural Resources Defense Council 
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Febru^y II, 2014 

The Hon. Lamar Smith 
Chair, Committee on Science, Space, 
and Technology 
2321 Rayburn HOB 
Washington, DC 205 15 

The Hon. David Schweikert 
Chair, Subcommittee on Environment 
2321 Rayburn HOB 
Washington, DC 20515 


The Hon. Eddie Bernice Johnson 
Ranking Member, Committee on Science, Space, 
and Technology 
2321 Rayburn HOB 
Washington, DC 20515 

The Hon. Suzanne Bonamici 

Ranking Member, Subcommittee on Environment 

232! Rayburn HOB 

Washington, DC 20515 


Dear Chairs and Ranking Members; 

I am writing in strong opposition to H.R. 40 1 2, the Secret Science Reform Act of 20 1 4. The 
legislation represents a solution in search of a problem that does not exist. The EPA already 
makes the data, methodology, and peer-reviewed research it relies on in its rule-making 
processes as transparent as possible. Moreover, the additional restrictions imposed by this 
proposed bill would make it almost impossible to base public protections on the best 
available scientific information. In particular, if enacted, the language appears to indicate that 
the agency would be inhibited by the following challenges: 

• The EPA wouldn’t be able to use most health studies. The agency would likely be 
prevented from using any study that uses personal health data. The confidentiality of 
such data is usually protected by institutional review boards (IRB); thus, the data 
could not be made publicly available as demanded. Since many EPA rules are health- 
based standards, this rule would severely restrict the ability of the agency to base 
rules on science. 

• The EPA wouldn’t be able to draw from industry data sources. The agency 
would be prevented from using data provided by industry to the agency. Since 
information from industry sources is often not publicly available, a law requiring as 
such would prevent the agency from utilizing industry data, a source of information 
that often provides otherwise unknown data to inform EPA rule-making. 

• The EPA wouldn’t be able to use new and innovative science. New scientific 
methods and data may be restricted by intellectual property protections or industry 
trade secret exemptions. This proposed bill would limit EPA’s ability to rely on the 
best available science including novel approaches that may not yet be publicly 
available. 
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• Long-term and meta analyses would be unavailable. Many of EPA’s heaith-based 
standards rely on long-term e)q)osure studies that assess the link between chronic 
diseases/mortality and pollutants; or on meta analyses that include many different 
studies and locations to provide a more robust look at the science. In H.R. 4012, the 
provision that studies be “in a manner that is sufficient for independent analysis and 
substantial reproduction of research” may prevent use of these vital studies by the 
EPA, as it is unclear whether such spatially and temporally comprehensive studies 
would be considered “sufficient for substantial reproduction.” 

1 strongly urge you to oppose the Secret Science Reform Act of 2014. The proposed bill 
would inhibit the EPA’s ability to carry out its science-based mission to protect human health 
and the environment 


Sincerely, 



Director, Center for Science and Democracy 


Union of Concerned Scientists 
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George Mason University Survey: 

Expert Opinion on Reguiatory Risk Assessment 


Introduction 

The challenges of assessing and regulating potential health risks to the public and the environment 
from chemical exposures have drawn much debate and growing interest from the expert community 
and Congress in recent years. The process and science behind how risk assessments are conducted 
is opaque to many people, because it involves assessing complex information from toxicology, 
pharmacokinetics, epidemiology, biostatistics and other areas. In addition, public controversy 
regarding the utility and quality of assessments increasingly accompanies decisions on the safety 
of individual chemicals as well as broader health assessment frameworks, such as the EPA’s IRIS 
process. 

Public debates on these matters typically involve representatives of industry and environmental 
groups, public officials, and individual scientists from a variety of institutions whose work bears 
on these issues. But there is no mechanism to tap into the collective opinion of the experts in these 
fields to capture their uniquely valuable insight. Moreover, members of the media sometimes 
focus attention on the loudest or most discordant voices, which may not be representative of 
informed opinion. 

It would be easier for the public and regulators alike to make informed decisions if they had some 
way of knowing the opinions of the broader expert community. The survey by the Center for 
Media and Public Affairs (CMPA) and Center for Health and Risk Communication (CHRC) at 
George Mason University described below was conducted to advance the current discussion over 
the role of risk assessment in government regulatory decisions by bringing the collective voice of 
the expert community into the public arena. 

To capture the viewpoint of the scientific community, we surveyed members of three professional 
organizations whose memberships represent repositories of knowledge and experience in 
regulatory risk assessment. They are the Risk Assessment Specialty Section of the Society of 
Toxicology (SOT-RASS), the Dose Response Section of the Society for Risk Analysis (SRA- 
DRS), and the International Society for Regulatory Toxicology and Pharmacology (ISRTP). 

Results 

The survey results outline the preferences of scientific experts for the conduct of regulatory risk 
assessments, as w'ell as their valuations of how well current procedures are working. We find 
general agreement on how elements of regulatory risk assessments should work, coupled with 
concern over how well they are working in practice. 




EXPERT OPINION ON REGULATORY RISK ASSESSMENT 


PAGEl 



245 


There is widespread endorsement of commencing assessments with problem formulations and 
analysis plans that are peer reviewed: data acquisition that includes aecess to raw data and the use 
of inclusion/exclusion criteria; applying data evaluation that uses the same criteria for evaluating 
studies regardless of institutional origin; implementing weight of evidence methodology that 
incorporates the use of non-linear (threshold) models; and establishing procedures insuring the 
independence and effectiveness of external peer review. 

By contrast, there is widespread concern over the current application of these procedures. Fewer 
than one in three scientists say that, in their experience, problem formulations are often conducted. 
Fewer than one in three affirm that raw data from critical studies are often made available to 
assessors or peer review'ers. Only one in four say that standardized search protocols are often used 
for collecting data. Fewer than half say that all relevant and reliable studies are often selected for 
evaluation, and only one in four say consistent and transparent criteria are often used to evaluate 
studies. 

Fewer than half affirm that weight of evidence methodology is often used, or tliat mode of action 
information is applied well for characterizing human risk. Only about one in four say that current 
peer review procedures provide sufificient input from stakeholders. 

Finally, respondents criticize the weighting of various factors in risk management. As they perceive 
current practices, too little attention is being given to scientific factors and economic costs and 
benefits, and too much attention is given to environmental groups, the precautionary principle, 
media coverage, and political concerns. 

The following is a summary of some of the key findings from the survey that touch on critical 
areas of developing assessments: 

Problem Formulation/ Analysis Plans 

Over two thirds of the experts (68 percent) believe it is “very important” to complete a 
problem formulation evaluation and have an analysis plan in place prior to conducting 
a regulatory risk assessment. However, fewer than half as many (30 percent) say that, 
in their experience, prior problem formulations were conducted. In addition, almost all 
respondents believe that analysis plans should be peer reviewed. Most (65 percent) regard 
an internal review as acceptable, while 34 percent think an external review is necessary. 
Only five percent say that no peer review of analysi.s plans is necessary. 

Data Acquisition 

A major element of risk assessments involves acquiring and evaluating evidence from 
studies that bear on the assessment. Most respondents (69 percent) regard it as “very 
impoifant” for assessors to have access to underlying raw data for the most critical studies 
in order to independently analyze their results. 
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However, only 3 1 percent report that, in their experience, such underlying raw data are 
"often” or "always” made available to assessors, while nearly as many (27 percent) say the 
data are “rarely” or “never” made available. (The remaining 42 percent say that the data are 
“sometimes” made available.) 

A somew'hat smaller majority (59 percent) see it as very important for peer review'ers as 
well to have access to underlying raw data from critical studies. In this case, only 16 percent 
report that this is done often or always, compared to 42 percent who say it is done rarely or 
never. 

One area in which there is almost universal agreement among these scientists concerns 
the use of inclusion/exclusion criteria for selecting the studies to be reviewed. Ninety-four 
percent support the use of such criteria, compared to only six percent who do not. 

However, only 24 percent report that, in their experience, standardized search protocols 
are often or always used and described for collecting all available study data. This is fewer 
than the 35 percent who say this was rarely or never done. 

Data Evaluation 

Respondents are less than sanguine with regard to the existing data evaluation process. 
While 44 percent say that the goal of using all relevant and reliable studies has often or 
always been met in risk assessments they are familiar with, 42 percent say this goal is met 
only sometimes, and 13 percent say it is met rarely or never. 

Similarly, fewer than one out of four respondents (24 percent) report that consistent and 
transparent criteria are often or always used to evaluate the quality and reliability of studies. 
Only 29 percent reported that such criteria are rarely or never used. 

Finally, there is widespread agreement (by 82 to 18 percent) that the same criteria should 
be used to evaluate the quality and reliability of all studies, regardless of their origin in 
academia, government, industry, contract labs, etc. 

Weight of Evidence 

We asked several questions regarding various aspects of the weight of evidence methods 
used to integrate various types of data in making an overall judgment on risk. Most 
respondents (89 percent) believe that weight of evidence methodology should be used, 
described, and documented for all risk assessments. 

How'ever, when asked how often, in their experience, weight of evidence methodology was 
applied in regulatory risk assessments, fewer than half (45 percent) replied that this was 
often or always used. Thirty-nine percent said it was sometimes done, and 1 6 percent report 
that it was rarely or never used. 
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When this methodology is used, only one out of four (24 percent) describe it as often or 
always consistent and transparent. About the same number (23 percent) describe it as rarely 
or never consistent and transparent. A slight majority (53 percent) say that it sometimes 
meets these criteria. 

When asked how well mode of action information is applied in characterizing risk to 
humans, only 39 percent replied that it is done well, compared to 61 percent who said it is 
done poorly. 

There was far greater agreement on the use of non-linear models and thresholds. When a 
non-mutagenic mode of action is indicated, 88 percent believe that non-linear (threshold) 
models should be used to estimate human risk from substances that cause cancer at high 
doses in lab animal studies. The same proportion would consider non-linear thresholds in 
mutagenic carcinogenesis as well. 

Similarly, when a threshold event is responsible for cancer effects, 82 percent would 
proceed with a linear low dose extrapolation, and nearly as many (75 percent) would do so 
in the absence of multiple tumor sites. 

Peer Review 

As with weight of evidence methodology, there is widespread agreement on the importance 
of external peer review in regulatory risk assessment. Seventy-three percent see external 
scientific peer review as very important and 24 percent see it as somewhat important. 

In addition, 78 percent believe the peer review process should be conducted independently 
of the office or program that develops a risk assessment. A smaller but still substantial 
majority (65 percent) would create an independent entity that insures authors would 
respond to peer review comments. 

In contrast to the agreement on the need for strong external peer review procedures, 
opinion is split over current practices. Only one out of four (25 percent) believe that current 
procedures often or always provide sufficient opportunity for input from stakeholders, 
compared to 3 1 percent who say this rarely or never happens. Even fewer (21 percent) say 
current processes assure that stakeholder input is thoroughly considered by peer reviewers, 
compared to 38 percent who say this happens rarely or never. 

Risk Management 

Two out of three respondents say they have taken part in formal discussions or reviews of 
risk management documents, and most are critical of the priorities and practices of risk 
management. Only 41 percent believe risk management decisions are adequately based 
on our current knowledge and understanding of biology and toxicology. In addition, they 
would change the relative weight they see risk managers as giving to various elements 
embedded in risk management decision-making. 
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Given a list of eight factors to choose from, respondents believe that risk managers 
currently give the greatest weight to the legal implications of regulatory decisions, followed 
closely by political concerns. These are followed by some closely clustered factors — ^the 
precautionary principle, environmental group concerns, scientific factors, media coverage, 
and economic costs and benefits. Industry concerns are perceived as receiving the lowest 
weight. 


By contrast, when respondents are asked how these same factors should be weighted, 
scientific factors far outpace all others. Economic costs and benefits finish a clear second, and 
legal implications an equally clear third. Then three factors are closely grouped together — 
industry concerns, environmental group concerns, and the precautionary principle. Lowest 
on the list are political concerns and media coverage, in that order. 


Methodology 

We created an online questionnaire with the assistance of Harris Interactive, a leading survey and 
market research firm and an industry leader in online polling. 

In February 2013, each organization sent a letter to its members inviting them to participate by 
accessing a link provided by Harris. A total of 186 respondents completed the questionnaire. This 
group included 167 members of SOT-RASS, 40 members of SRA-DRS, and 27 members ofISRTP. 
(These numbers reflect membership in more than one organization by some individuals.) Response 
rates were 23 percent for SOT-RASS, 28 percent for SRA-DRS, and 27 percent for ISRTP, 

The questionnaire addressed attitudes toward several important aspects of regulatory risk 
assessment. These include the use of problem formulation and accompanying analysis plans, the 
acquisition and evaluation of data, the application of weight of evidence methodology, the role of 
peer review, and the use of/adherence to guidance documents. 

In addition, we asked respondents for their opinions about factors involved in risk management 
decisions. (Most reported taking part in formal risk management reviews as well as in risk 
assessments.) Finally, we inquired about various background factors, such as their occupation, 
discipline, certification, experience, and area of expertise. These sample characteristics are listed 
as an addendum below. 

Addendum: Sample Characteristics 

Tile sample is 58 percent male and 42 percent female and averages 54 years of age. Over 80 
percent have a PhD, and 58 percent possess a professional certification, led by 45 percent who are 
certified by the American Board of Toxicology. 

The sample is relatively diverse in occupations and expertise. Respondents’ primary area of expertise 
is spread among 3 i percent who are primarily expert in risk characterization, 28 percent in hazard 
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identification, 24 percent in dose-response assessment, and 10 percent in exposure assessment. 
Thirty-one percent work in industry, 30 percent are consultants, 25 percent are employed by 
government entities, and 13 percent are based in academia or non-profit organizations. 

Respondents also report considerable and diverse experience in areas of concern. Over 80 percent 
have worked on industrial chemicals, over 60 percenton pesticides and water contaminants, over 50 
percent on consumer products and occupational risks, over 40 percent on air pollution, hazardous 
waste, and food products, and over 30 percent on pharmaceuticals. In addition, over two-thirds say 
they have worked in their field for 20 years or more. 

This group also has widespread experience with risk assessment. Just under half (49 percent) 
have developed risk assessments for government agencies, and 77 percent have contributed to 
or reviewed government risk assessments. In addition, nearly two-thirds (65 percent) have 
developed risk assessments and 75 percent have contributed to or reviewed risk assessments for 
non-government entities. 

In view of the qualifications and experience reported by respondents, it seems appropriate to regard 
them as representing an expert community with regard to risk assessment. 

For more information on the professional organizations referred to, please see: 

The Society of Toxicology: www.loxicology.org 
The Society for Risk Analysis; www.sra.org 

The International Society of Regulatory Toxicology & Pharmacology: www.isrtp.org 
The American Board of Toxicology : www.abtox.org/HomePage.aspx 


Funding for this survey was provided by the American Chemistry Council, Crop Life 
America, and the International Platinum Group Metals Association. 
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ffl STATS 

take a quantitative leap 


Center for Media and Public Affairs & 
The Statistical Assessment Service 
George Mason University 

933 N. Kenmore St, Suite 405 
Arlington, VA 22201 


Overview: CMPA Regulatory Risk Assessment Survey 


Problem Formulation/Analysis Plans 

Percent 

Problem formulation/analysis plan very 
important 

68 

Problem fomiulation always/often 
conducted 

30 

Analysis plan should be peer reviewed 

95 

External review necessary 

34 


Data Acquisition 

Percent 

Access to raw data by assessors very 
important 

69 

Raw data made available to assessors 
often/always 

31 

Access to data by peer reviewers very 
important 

59 

Raw data made available to peer reviewers 
often/alwavs 

16 

Inclusion/exclusion criteria should be used 

94 

Standardized search protocols are used 
oftcn/always 

24 


Data Evaluation 

Percent 

Goal of using all relevant and reliable 
studies is met always/often 

44 

Consistent/transparent criteria are used to 
evaluate studies always/often 

24 

Same criteria should be used to evaluate 
studies of all origins 

82 
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Weight of Evidence 

Percent 

Should use weight of evidence method for 
all risk assessments 

89 

Weight of evidence methodology used 
often/alwavs 

45 

Weight of evidence approach 
consistent/transparent often/always 

24 

Mode of action info applied somewhat/very 
well 

39 

Should use non-linear model to estimate 
human risk from animal studies 

88 

Non-linear thresholds should be considered 

88 

When threshold event responsible for 
cancer, next step is; 


If multiple tumors, use non-linear low dose 
extrapolation 

75 

If absence of multiple tumors, use non- 
linear low dose extrapolation 

82 


External Input 

Percent 

External peer review is very important 

73 

Review should be independent of office/ 
program that developed the assessment 

78 

Should create independent entity to ensure 
authors respond to review 

65 

Process often/always provides opportunity 
for stakeholder input 

25 

Input from experts, public thoroughly 
considered often/alwavs 

21 


Guidance Documents 

Percent 

Served as peer reviewer for regulatory risk 
assessment 

53 

If so, very knowledgeable about guidance 
documents used 

63 

Government agencies follow own guidance 
documents often/always 

51 
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Risk Management Factors 

• Risk management deeisions are based on current knowledge of biology and 
toxieology very/somewhat well. 41% 


How much weight do risk managers currently give to: 


Risk Management Factors 

Great Deal of Weight (%) * 

Mean Score (1-5) 

Legal implications 

72 

3.9 

Political concerns 

66 

3.8 

Precautionary principle 

52 

3.4 

Environmental groups 

49 

3.4 

Science 

47 

3.5 

Media coverage 

43 

3.3 

Economic costs/benefits 

41 

3.2 

Industry 

28 

2.9 


How much weight should risk managers give to: 


Risk Management Factors 

Great Deal (%)* 

Mean Score (1-5) 

Science 

98 

4.8 

Economic costs/benefits 

67 

3.9 

Legal implications 

48 

3.5 

Industry 

20 

3.0 

Precautionary principle 

19 

2.5 

Environmental groups 

16 

2.8 

Political concents 

8 

2.1 

Media coverage 

4 

1.8 


* 4 or 5 on scale from l=none to 5=great deal Mean seorc on 1 to 5 scale from l=none to 
5=great deal 
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Respondent Profile 


Background 

Percent 

Male 

58 

Average age 

54 

Ph.D. 

82 

Years in field 20+ 

68 


Field 

Percent 

Toxicoiogv/phanuacology 

65 

Enviro Health 

8 

Cell/molecular biology 

5 


Certification 

Percent 

American Board of Toxicology 

45 

Academy of Toxicological Sciences 

13 


Occupation 

Percent 

Industry 

31 

Consulting 

30 

Government 

25 

Academic/non-profit 

13 


Primary RA expertise 

Percent 

Risk characterization 

31 

Hazard identification 

28 

Dose-response assessment 

24 

Exposure assessment 

10 
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Areas worked in 

Percent 

Industrial chemicals 

83 

Pesticides 

63 

Water contaminants 

60 

Consumer products 

53 

Occupational 

52 

Air pollution 

47 

Hazardous waste 

46 

Food 

42 

Pharmaceuticals 

31 


Risk Assessment Experience 

Percent 

Developed gov't RA’s 

49 

Developed non-gov’t RA’s 

65 

Contributed to/reviewed gov’t RA’s 

77 

Contributed to/reviewed non-gov’t RA’s 

75 

Took part in formal risk management 
reviews 

66 
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Executive Summary 


The use of science in the formulation of regulatory policy - by both the Executive Branch and the Congress - has been 
a political flashpoint in recent decades.' Policy makers often claim that particular regulatory decisions have been driven 
by, or even required by science; their critics, in turn, have attacked the quality or the interpretation of that science. Such 
conflict has left the U.S, with a system that is plagued by charges that science is being “politicized" and that regulation 
lacks a solid scientific basis. As a result, needed regulation may be stymied, dubious regulations may be adopted, issues 
can drag on without conclusion and policy debate is degraded. Moreover, the morale of scientists is weakened, and 
public faith in both government and science is undermined. 

The question is not whether scientific results should be used in developing regulatory policy, but how they should be 
used. This report is structured around three sets of questions that are at the heart of the debate over the use of science in 
regulatory policy. Those questions are: 


■ What kinds of activities or decision-making amount 
to "politicizing'’ science? How and to what extent can 
one differentiate betv^-een the aspects of regulatory 
policy that involve scientific judgments and those 
that involve making policy recommendations (which 
are inherently political)? 

■ When and how should Federal agencies empanel 
advisory committees? How should members be 
selected? How should conflicts of interest and 


biases of potential members be handled? What is 
scientific balance and how can it be achieved? How 
can the independence and integrity of committees' 
deliberations be assured? 

■ What studies should agencies and advisory 

committees review in formufating regulatory policy? 
How should they be weighed? What role should 
peer review play and how might peer review be 
modified and strengthened? 


Implementing our panel's answers to these questions would result, we believe, in a more candid, transparent, 
and rigorous use of science in regulatory policy making and a more honest and thoughtful debate about 
regulatory proposals.- 


In ;h «; repor: 'sc nc!.’' fc'eis namoi and physical sciences and engineering, 

"hi’ pi itnarv (onus or tf-is report is tl!c sw'snicpicbiems affecting the role of science in reguiatcry policymakir^g. sudt as conflating science gcesnons and pot-cy 
questions as they a-e ic na srenial lor.qstarsding dr>3 often overlooked. Other {«ms of misuse or abuse of sdence in government are .ii i roi,ii» a, («=- ■'i, d 

cv [nis panel s< |p.?ra<:niorition:nCh’piei' ! that 'pditical decision-makers sivauld never dictate what scientilk studies should cend^d' ”Hcv/r r t-n n s- jfs 
bi‘cndis<;;,<»(>deisyi.v'ic‘te andarcncttheptiiri3rysubjectofthisreport,Evenife.erypotenti3iabuseofscierKewereavoy(?d,ic-gu!«'''<»rc ds-M i '■ r-’din 

avn:cl?b;p clebsrns unless -he ssues d scussed in this repevt were addressed. 
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Inipi'ovinsi the Lhc ot Science in Regulatory Policy 


Recommendations 

Each chapter of the report makes an overarching 
recommendation (those numbered below) and then 
elaborates on how to implement it. Those more detailed 
recommendations, listed below, are generally in bold in the 
text of the report. 


B RECOMMENDATION ONE: The Administration 
needs to promulgate guidelines (through 
executive orders or other instruments) to ensure 
that when federal agencies are developing 
regulatory policies, they explicitly differentiate, 
to the extent possible, between questions that 
involve scientific judgments and questions that 
involve judgments about economics, ethics and 
other matters of policy. 


Poiitica! decision-makers should never dictate what 
scientific studies should conclude, and they should 
base policy on a thorough review of all relevant 
research and the provisions of the relevant statutes. 

But some disputes over the "politicization' of science 
actually arise over differences about policy choices that 
science can inform, but not determine. 

The Administration needs to devise regulatory 
processes that, in as many situations as possible, 
could help clarify for both officials and the general 
public which aspects of disputes are truly about 
scientific results and which concern policy. At a 
minimum, the Administration should require that a 
section of the Federal Register notice for any proposed 
guidance or rule that is informed by scientific studies 
describe the primary scientific questions and the 
primary policy questions that needed to be answered 
in drafting the rule. 



An additional approach to clarifying this distinction 
would be to also require the Federal Register notice to 
include answers to such questions as; What additional 
science would change the debate over a proposed 
regulatory policy and in what ways would the debate 
change? Another possible approach would be to 
require federal agencies to spell out genuine alternative 
regulatory policies when proposing guidance or a rule, 

The first impulse of those concerned with regulatory 
policy should not be to claim "the science made me 
do it" or to dismiss or discount scientific results, but 
rather to publicly discuss the policies and values 
that legitimately affect how science gets applied in 
decision making. 
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B RECOMMENDATION TWO: The Administration 
should promulgate guidelines (through executive 
orders or other instruments) directing agencies 
to follow the policies described below on: when 
to consult advisory panels on scientific questions, 
how to appoint them (including how to deal with 
conflicts of interest and biases), and how they 
should operate. Congress should pass, and the 
President should sign into law, any statutory 
changes needed to implement these policies. 

Federal agencies should use scientific advisory 
committees to the maximum extent possible to review 
the science behind regulatory policies. 

TYPES OF ADVISORY PANELS 

Advisory committees that are to exclusively address 
science questions (referred to in this report as "scientific 
advisory committees") should generally consist only of 
members with relevant scientific expertise. 

All non-government members of scientific advisory 
committees should be appointed as Special 
Government Employees. 

In general, scientific advisory panels should not be 
asked to recommend specific regulatory policies. 

The remainder of the recommendations are concerned 
exclusively with procedures related to scientific 
advisory panels. 


TRANSPARENCY IN THE SELECTION PROCESS 

The process of naming advisory committee members 
should be made more transparent. Options for 
achieving greater transparency include: seeking 
recommendations for members on the Web and/ 
or through contacts with relevant groups; publicly 
announcing on the Web the criteria for membership 
(such as the range of scientific disciplines that need to 
be included); and announcing proposed members on 
the Web to solicit public comment. 

FACTORS IN SELECTING ADVISORY 
COMMITTEE MEMBERS 

The primary purpose in appointing a committee is 
to gather a group of eminently qualified individuals 
w'ho can have an open, engaged and comprehensive 
discussion of the issues before them. Appointing a 
committee capable of comprehensive discussion 
involves, among other things, achieving balance 
among the applicable scientific disciplines. Moreover, 
agencies should avoid turning repeatedly to the same 
scientists for service on advisory committees. And 
agencies should periodically turn over the staff that is 
assigned to select panelists. 

DISCLOSURE OF QUALIFICATIONS, 

FINANCES AND ACTIVITIES 

Members of federal scientific advisory committees 
should be required to disclose to the government 
information on relevant financial relationships 
and professional activities (such as giving talks at 
conferences and testifying in court) going backfive 
years. Members should also be asked to disclose, to the 
best of their ability, any relevant professional activities 
that occurred more than five years prior to their 
committee service. 
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Federal agency disclosure forms should be as clear 
and uniform as possible, Developing a single form 
that draws on the different forms used now by the 
Environmental Protection Agency (EPA) and the Food 
and Drug Administration (FDA) would be a good start. 

For financial disclosure, the categories of information 
the National Academy of Sciences views as relevant for 
its panelists are also appropriate for federal agencies. 

To build public trust through transparency, much 
more information on federal advisory committee 
members needs to be available than is now the 
case. One possibility would be for federal agencies 
to make publicly available all the information on a 
panelist's disclosure form except the precise dollar 
amounts of their stock holdings or compensation 
and any information on the finances of their spouse 
or dependent children. At the same time, the agency 
would disclose the member's educational background 
and scientific credentials. 


DETERMINiNG CONFLICTS OF INTEREST 

For conflia of interest, there must be a clearer federal 
policy with bright lines that leaves as little doubt as 
possible as to who would be considered to have a 
conflict if they served on a particular advisory committee. 
The definition should be as uniform across agencies as 
possible and, at the very least, should set a minimum 
standard for all agencies. The general principles that the 
National Academyof Sciences uses to define conflict 
of interest apply equally well to the government. The 
question to be asked in defining a conflict of interest is 
whether a particular financial relationship would tend 
to constrain a generic individual's point of view. Such 
relationships need to be defined as conflicts regardless of 
the source of the funding. 

When considering whether a conflict of interest exists, 
federal agencies should look back two years rather 
than just considering current relationships as is now 
the case. Two members of our panel dissented from the 
recommendatiorj in this paragraph for reasons described 
on page 22. 

DEALING WITH CONFLICTS OF INTEREST 

nie desired norm for federal agencies should be to 
appoint advisory committees whose members are free 
of conflias of interest. There will be instances, though, 
when scientists with conflicts of interest may be 
needed for a panel because of their expertise. 

The standard for allowing someone with a conflict of 
interest to serve on an advisory committee should be 
changed to the clearerand arguably more stringent 
policy of the National Academy of Sciences under which 
a conflicted expert can serve only in a situation where 
having a confliaed panel member is "unavoidable." !Jhe 
current standard Is whether the need for the conflicted 
member's services 'outweighs the potential for a conflict 
of interest posed by the financial interest involved.'') 
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Appointment of an individual with a conflict of 
interest should require a forma! waiver from the 
appointing official. When a waiver is granted, the 
agency should publicly state that the appointee has 
a conflict and should provide enough information 
that the public and the other committee members 
understand what kinds of efforts were made to find a 
non-conflicted individual, how and why the appointed 
individual was considered to be conflicted, and why 
the individual was appointed nonetheless, as well as 
disclosing who signed off on the waiver. 

Agencies should not appoint anyone with a conflia 
to serve as the chair or co-chair of a committee. And 
agencies should limit The issuance of conflict waivers. 

DETERMINING AND DEALING WITH BIAS 

The federal government should follow the National 
Academy's lead and distinguish clearly between conflict 
ofirtterestand bias, 

The goal should generally be to assemble committees 
of individuals who are as impartial (i.e., fair-minded) as 
possible and to ensure that the overall committee is bal- 
anced. Agencies should not shy away from Including 
scientists on a panel who are considered "outliers" on the 
question(s) under consideration, provided that the scien- 
tist is a respected practitioner in a relevant field and the 
committee as a whole fairly represents the mainstream. 


Committee members need to know of each other's 
financial relationships and viewpoints. Moreover, the 
appropriate agency official needs to take an active role 
in supporting the committee's work, which includes 
managing any conflicts for the duration of the panel. 

Federal agencies should not be able to circumvent 
the processes discussed above by contracting out the 
appointment or operation of advisory committees. 

The Administration and the Congress should carefully 
think through the benefits and disadvantages of 
requiring alt meetings to be open, It might be worth 
considering, for example, whether some scientific 
advisory committees could be allowed to hold some 
closed meetings if the selection process for committee 
membership were more open than it generally is today 
(as recommended above). 

The recommendations of a committee, though, 
must always be made public (assuming no classified 
information is involved), and indeed committees should 
be required to explain fully their methodology and the 
rationale for tMr concluslonsAn the Federal Register notice . 
for any rule for which a scientific advisory committee was 
convened, the federal agency should be required to state 
whether it differed with any conclusions ofa .scientific 
advisory committee and if so, why, and should be required 
to explain how the new regulatory policy is consistent 
with the conclusions that were accepted. 


MANAGING ADVISORY COMMITTEES 

Once the final members of a committee have been 
named, federal agencies need to defend their choices 
of appointees and stand by their panel if it comes 
under attack. 
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a RECOMMENDATION THREE: Agencies and their 
scientific advisory committees should cast a wide 
net in reviewing studies relevant to regulatory 
policy, and should make their methods for filtering 
and evaluating those studies more transparent. 


TRANSPARENCY 

The process of conducting literature reviews should 
become more transparent Agencies and their scientific 
advisory committees should be explicit about the 
criteria they are using to determine which scientific 
papers to review and how those papers are being 
evaluated. Once an agency has opened a docket on a 
rule or guidance that will draw on scientific studies, it 
should make available on the Web a list of the studies it 
is reviewing and should regularly update the list. 

CRITERIA 

In general, papers in high impact, peer reviewed journals 
should be given great weight, and papers that have not 
been peer reviewed should be treated with skepticism. 
But agencies and scientific advisory commiirtees need to 
extend their inquiry beyond simply ascertaining whether 
a paper was peer reviewed; peer review is a necessary but 
not sufficient determinant of quality. Conversely, sRidies 
that have not been peer reviewed should not be sum- 
marily rejected ifthey appear to contribute to the inquiry. 

in general, agencies and scientific advisory committees 
should be wary of studies when it is unclear 
who funded the study or whether the principal 
invesiigaTor(s) had any conflicts of interest. 

Agencies and scientific advisory committees should be 
extremely skeptical of a scientific study unless they are 
sure that the principal investigatorfs) (as opposed to the 
sponsor or funder) had ultimate control over the design 
and publication of the study. 


In fields where a public registry of studies exists {such as 
the registry established by the Food and Drug Adminis- 
tration Modernization Act of 1 997), agencies and scien- 
tific advisory committees should consider the relevant 
registered studies and should be wary of studies that 
met the criteria for the registry, but were not registered. 

DATA AVAILABILITY 

Studies used in the formulation of regulation should be 
subject to data access requirements equivalent to those 
under the Data Access Act (Shelby Amendment) and its 
implementing circular regardless of who funded the study. 

Confidential Business information (CBI) is a legitimate 
and needed designation for information submitted to 
the federal government, but it appears to be overused 
today. The Administration and the Congress should 
gather data on the extent and nature of CBI claims, The 
Administration and Congress should consider requiring 
each new CBi claim to include a brief, but substantive 
justification for the claim. 

ADDITIONAL STUDIES 

Agencies should experiment with a variety of additional 
approaches that would enable them to commission 
studies and literature reviews related to pending regu- 
latory decisions that would be widely seen as unbiased, 

PRESENTING CONCLUSIONS 

In presenting the conclusions of literature reviews, agen- 
cies and their scientific advisory committees need to be 
as open and precise as possible in discussing levels of risk 
and uncertainty. Policy makers should be wary of conclu- 
stons about risk that are expressed as a single number. 





B RECOMMENDATION FOUR: The federal 
government, universities, scientific journals 
and scientists themselves can help improve 
the use of science in the regulatory process 
by strengthening peer review, expanding the 
information available about scientific studies, 
and setting and enforcing clear standards 
governing conflict of interest. 


PEER REVIEW 

Federal agencies need to experiment with ways to 
increase the number of scientists who participate 
in peer review, particularly peer review of draft 
manuscripts. Universities should do more to 
make service as a peer reviewer an expected and 
appreciated aspect of a scientist's career. Scientific 
journals should improve the quality control of peer 
review and should experiment with different ways 
of conducting peer reviews. The report lists a number 
of specif)c approaches that could be tried by the 
governmen t, universities and journals. 



INFORMATION ON SCIENTIFIC STUDIES 

Federal agencies, universities and journals should 
encourage or require on-line publication of the methods 
and data underlying published scientific studies. 

Federal agencies should determine whether the 
idea of research registries, which today is focused on 
research related to pharmaceuticals, can be expanded 
toother fields. 

CONFLICT OF INTEREST 

Journals should have clear, publicly accessible 
conflict-of-interest policies and should require full 
disclosure of how studies were funded and of any and 
all conflicts of interest they determine an author has, 
Editors should also disclose any of their own conflicts 
of interest. In addition, journals should consider 
requiring authors to certify that they had ultimate 
control over the design and publication of the study 
being described in a paper. 

Federal agencies need to consider promulgating rules 
that would sanction scientists who run afoul of federal, 
university or journal requirements concerning disclosure, 
conflict of interest or ultimate sponsor control. 
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Introduction 


The use of science in the formulation of regulatory 
policy - by both the Executive Branch and the Congress 
- has been a political flashpoint in recent decades.' 
Policy makers often claim that particular regulatory 
decisions have been driven by, or even required by 
science; their critics, in turn, have attacked the quality or 
the interpretation of that science. Such conflict has left 
the U.S. with a system that is plagued by charges that 
science is being "politicized" and that regulation lacks 
a solid scientific basis. As a result, needed regulation 
may be stymied, dubious regulations may be adopted, 
issues can drag on without conclusion and policy 
debate is degraded, Moreover, the morale of scientists 
is weakened, and public faith in both government and 
science is undermined. 

These problems are largely systemic; they will not 
magically vanish with a change of Administrations or 
a shift in the composition of the Congress.^ But the 
advent of a new Administration and a new Congress 
is an opportune time to take stock of the situation and 
to try to devise ways to get beyond the predictable 
battles that would otherwise lie ahead. The use of 
science in regulatory policy is another area in which 
government needs to get beyond the stale debates 


i'i ihis refxx t, 'sc'cice' refers to the-'isiorei and p.nys^t sciences ar>d engineering. 
' '’hi? primary focus of tnis report Is the systemic. pvoWrtmsaffectinci the role 
sc'ence in regulatory jwl cyrnaki",g, such as cwifiating science questions and 
policy Questions, as they arc fundamental, tongsianding and often overlooked. 
Otner forms of rtususe or aOuse of sciexc inQcv«nment are also unacceptabte, 
as reflected t'y tnis oanol s clear acimaniiori in Chapter 1 that ‘potSical decision- 
nt.skers shi.'.Liid never dictate wr.at sc-emific studies should corKliide.*Hcv.'eve<. 
‘'St Sue ch tno L ssc" t, s,.ycheic, and are not the primary sut^t 

r -i 5 p vn tv ryocti ?l abuse of science were avoided, leguiatcfv 

p Is c n c dtL edebatesunless theissiiesdiscussed in 

this report adijfossed. 


and false dichotomies of the past. The question is not 
whether scientific results should be used in developing 
regulatory policy, but how they should be used. 

New governmental processes are needed - approaches 
that will be seen as legitimate by stakeholders on ail 
sides of issues and that will make policy making more 
transparent. A critical goal of any new procedures for 
establishing regulatory policy must be to clarify which 


A tendency to frame regulatoqr 
issues as debates solely about science, 
regardless of the actual subject in 
dispute, is at the root of the stalemate 
and acrimony all too present in the 
regulatory system today. 


aspects of a regulatory issue are matters of science and 
which are matters of policy (e.g., economics or ethics). 

A tendency to frame regulatory issues as debates 
solely about science, regardless of the actual subject in 
dispute, is at the root of the .stalemate and acrimony all 
too present in the regulatory system today. 

To come up with new approaches, the Bipartisan Policy 
Center established the Science for Policy Project, To carry 
out the project the Center assembled a diverse pane! 
of experts to develop recommendations for both the 
Executive Branch and the Congress on how to improve 
the way science is used in making regulatory policy 
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across the government's areas of responsibility. The 
pane! includes liberals and conservatives, Republicans 
and Democrats, scientists and policy experts, and leaders 
with experience in government, industry, academia and 
non-governmental organizations. 

This report is Structured around three sets of questions 
that are at the heart of the debate over the use of 
science in regulatory policy. (By "regulatory policy,' 
we mean not only specific rules, but all regulatory 
statements and guidance issued by Administration 
officials, and statements, hearings and legislation from 
the Congress.) Those questions are: 

■ What kinds of activities or decision-making amount 
to "politicizing" science? How and to what extent can 
one differentiate between the aspects of regulatory 
policy that involve scientific judgments and those 
that involve making policy recommendations (which 
are inherently political)? 

■ When and how should Federal agencies empanel 
advisory committees? How should members be 
selected? How should confiicts of interest and 
biases of potential members be handled? What is 
scientific balance and how can it be achieved? How 
can the independence and integrity of committees' 
deliberations be assured? 

■ What studies should agencies and advisory 
committees review in formulating regulatory policy? 
How should they be weighed? What role should 
peer review play and how might peer review be 
modified and strengthened? 

Implementing our panel's answers to these questions 
would result, we believe, in a more candid, transparent, 
and rigorous use of science in regulatory policy making 
and a more honest and thoughtful debate about 
regulatory proposals. 


With those results in mind, our hope is that this report 
will help shape, among other things, the implementation 
of the President's scientific integrity memorandum, 
which raises questions similar to those above.^ 

Like the Presidential memorandum, this report 
does not focus on any particular area of regulatory 
policy. Instead, we recommend principles and 
procedures that we believe should improve regulatory 
policymaking and debate across the board. Our 
recommendations are focused on the procedures 
used by the regulatory agencies. 

But while ouf only specific recommendation for 
Congress is to make any statutory changes needed 
to implement our proposals regarding scientific 
advisory committees (see Chapter 2), this report 
has additional implications for the Legislative 
Branch. Like the Executive Branch, Congress needs 
to take to heart, and find ways to implement, the 
principles and processes the report recommends, 
More specifically, Congress needs to find ways to 
distinguish the aspects of regulatory policy that 
involve scientific judgments from those that involve 
making policy recommendations in its debates, in the 
questioning of its witnesses, and in its parsing of the 
arguments brought to it by the Administration and 
lobbyists on all sides of regulatory issues. In addition. 
Congress should consider codifying the principles 
and procedures recommended in this report to make 
them less likely to change from Administration to 
Administration. Congress could consider passing 
general legislation on the issues in this report and/ 
or including provisions in legislation as relevant 


,V5e!r<xandum for the Heads of uxecutw Aqe.’ic'e:;, Mjrcn 9, 

2009.hnpi<Vedod«i3ccess«po,Qov/2009/pdi.'>-9-i44.i pdf. tt siwuid be .taed 
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programs are reauthorized. In doing so, however. 
Congress should avoid becoming overly prescriptive. 

Again, our firmest and most fervent hope is that 
this report will help point the Administration, the 
Congress, the media, interest groups and the courts 
to think more carefully and to speak more precisely 
about what is truly at issue when regulatory proposals 
are being debated, 

This report proposes specific procedures that we 
believe can inaugurate a new era - an era in which 
the science behind regulatory proposals will emerge 


from a more transparent and credible process that fully 
acknowledges the complexifies of reaching scientific 
conclusions; and in which the disagreements over 
political ideology, economics and values that are at 
the heart of many regulatory disputes will be debated 
openly and fully, not transmogrified into a political 
battle waged through science. 

This new era will not come into being, and certainly will 
not be sustained, merely by public officials claiming to 
mean well or trying to do their best. Change requires 
institulionaliidng specific procedures that will inculcate 
and direct this new way of thinking, This report 
recommends just such procedures, 

if our recommendations are implemented and succeed 
as we hope, science will be better protected and 
political values will be more fully debated, enhancing 
the process of regulatory policy making, and ultimately, 
democracy itself. The result should be better regulatory 
policy that protects the public both from needless 
regulations and from needless dangers. 


13 
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Chapter One 


RECOMMENDATION ONE: The Administration needs to promulgate 
guidelines (through executive orders or other instruments) to ensure that 
when federal agencies are developing regulatory policies, they explicitly 
differentiate, to the extent possible, between questions that involve scientific 
judgments and questions that involve judgments about economics, ethics and 
other matters of policy. 


Political decision'inakers should never dictate what 
scientific studies should conclude, and they should 
base policy on a thorough review of all relevant 
research and the provisions of the reievant statutes. 
But some disputes over the "politicization" of 
science actually arise over differences about policy 
choices that science can inform, but not determine. 
For example, decisions about how much risk society 
should tolerate or what actions should be taken in the 
face of scientific uncertainty are not science questions, 
rather they concern policies and values. Matters such as 
risk and uncertainty need to be informed by scientific 
results, but science cannot tell policy makers how to 
act, True, distinguishing between science and policy is 
not always easy or straightforward, and scientists may 
make choices based on values in the course of their 
work. Nonetheless, policy debate would be clarified 
and enhanced if a systematic effort were made to 
distinguish between questions that can be resolved 
through scientific judgments and those that involve 
judgments about values and other matters of policy 
when regulatory issues comprise both. This transparency 
would both help force values debates into the open and 
could limit spurious claims about, and attacks on science. 


It would also help policy makers determine which 
experts to turn to for advice on regulatory questions, 
and what kinds of questions they should be expected to 
answer, 

The Administration needs to devise regulatory 
processes that, in as many situations as possible, 
could help clarify for both officials and the general 
public which aspects of disputes are truly about 
scientific results and which concern policy. That 
distinction also needs to be spelled out in regulatory, 
documents. At a minimum, the Administration should 
require that a section of the Federal Register notice 
forany proposed guidance or rule that is informed 
by scientific studies describe the primary scientific 
questions and the primary policy questions that 
needed to be answered in drafting the rule. For 
example, for a clean air rule, the scientific questions 
might include how many excess deaths or hospital 
admissions would be expected to result from different 
atmospheric concentrations of the pollutant. The policy 
questions would include how to decide what level of 
concentration to allow, given the scientific information. 
The Federal Register notice would go on to describe 
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the answers to the listed questions and their rationale. 
(Chapters 2 and 3 discuss how federal agencies should 
obtain and characterize answers to scientific questions.) 

One approach that could help clarify the often 
problematic distinction would be to also require the 
Federal Register notice to include answers to such 
questions as: What additional science would change 


Unless clarifying science and policy 
issues becomes a central aspect of 
regulatory policy discussions, it 
will be veiy difficult to get beyond 
the finger-pointing and misleading 
debates that have been a banier to 
sensible policy making for so long. 


the debate over a proposed regulatory policy and 
In what ways would the debate change? This both 
would help pinpoint the nature and extent of scientific 
uncertainty and would highlight which aspects of a 
regulatory issue are not primarily about science. 

Another possible approach would be to require 
federal agencies to spell out genuine alternative 
regulatory policies when proposing guidance or 
a rule. Although this approach is embodied in some 
federal decision processes (e.g., those under the 
National Environmental Policy Act), the approach is not 
uniformly applied, and the alternatives proposed can 
be less than genuine. The idea would be to make clear 
the range of policy options that were available, given 
the science and the requirements of law. For example, 
agencies could be required to describe alternatives of 
different levels of stringency (or cost, when allowed by 
statute) that would be in keeping with the science and 
would comply with statutory mandates. 


Many additional options for implementing 
Recommendation One might be developed, but the 
goal should be to change the conversation about 
regulation and to inculcate new habits of thought. The 

first Impulse of those concerned with regulatory 
policy should not be to claim "the science made me 
do it" or to dismiss or discount scientific results, but 
rather to publicly discuss the policies and values 
that legitimately affect how science gets applied in 
decision making. 

No system for clarifying the roles of science and policy 
questions in regulatory decision making will be air tight 
or completely immune from abuse, But that is not a 
reason to adhere to the status quo, Unless clarifying 
science and policy issues becomes a central aspect of 
regulatory policy discussions, it will be very difficult to 
get beyond the finger-pointing and misleading debates 
that have been a barrier to sensible policy making for 
so long, in short, there must be clarity and transparency 
about the roles of policy and science in regulatory 
decisions for science to be appropriately integrated in 
regulatoty policy. 
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Chapter Two 


RECOMMENDATION TWO: The Administration should promulgate 
guidelines (through executive orders or other instruments) directing agencies 
to follow the policies described below on: when to consult advisory panels 
on scientific questions, how to appoint them (including how to deal with 
conflicts of interest and biases), and how they should operate. Congress 
should pass, and the President should sign into law, any statutory changes 
needed to implement these policies. 


Federal agencies should use scientific advisory 
committees to the maximum extent possible to 
review the science behind regulatory policies. (At the 

same time, agencies should be working to strengthen 
the internal capabilities of their staffs, including 
their scientists,’) Public officials should not delegate 
their ultimate responsibility to set policy to advisory 
committees. But scientific advisory committees can help 
ensure that policies are based on a range of scientific 
knowledge and perspectives, and they can make the 
regulatory process more transparent. As a result, the 
proper use of advisoiy committees can make it easier to 
adopt and more difficult to overturn good regulations. 

Types of advisory panels 

The first questiof) for an agency establishing an advisory 
committee should be whether the committee's charge 
will be to handle science questions or policy questions 
(or perhaps both). Agencies should ensure that science 
and policy questions are distinguished as dearly as 


Aococ'M niav want to i.'rri|:3nel adviwryccnrrvrtces to prcwie recomroaidations 
on how ;o strtnoirit'n the r nteios! expcihse, bo! fi^ismpca deals only with acIviKsy 
ccmrn ticcs chsraea won snsvvcfina questions pt^iitiing to regulatoiy poTxiy. 


possible in charges to advisory panels. Advisory 
comminees that are to exclusively address science 
questions (referred to in this report as "scientific 
advisory committees") should generally consist 
only of members with relevant scientific expertise. 
Advisory committees that are to address policy questions 
that are informed by science should include members 
with relevant scientific expertise along with policy 
specialists and stakeholders. 

All non-government members of scientific advisory 
committees should be appointed as Special 
Government Employees to ensure they comply with 
the appropriate ethics guidelines and requirements. 

In general, scientific advisory panels should not be 
asked to recommend specific regulatory policies.^ 


CXi( pane: is no: arguing that sclent sts snciii-d never O!’ consu led on po <(:y 
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Rather, they should be empaneled to reach conclusions 
about the science that would help guide a regulatory 
policy decision. They might also be charged with 
evaluating a regulatory option or options developed 
by federal officials in light of current scientific 
understanding, For example, a scientific advisory 
corrimittee might be asked to determine if a proposed 
standard was consistent with achieving a level of risk 
prescribed by federal officials.^ 

PLEASE NOTE: the remainder of this chapter is concerned 
exclusively with procedures related to scientific 
advisory panels. 

Transparency in the selection process 

The process of naming advisory committees 
should be made more transparent. Options for 
achieving greater transparency include; seeking 
recommendations for members on the Web and/ 
or through contacts with relevant groups,-^ publicly 
announcing on the Web the criteria for membership 
(such as the range of scientific disciplines that need 
to be included); and announcing proposed members 
on the Web (along with their disclosure forms, to the 
extent discussed below) to solicit public comment 
Agencies would then respond to the comments 


T-liefe are some cases in which a scit'otific deiesminetion fliiiomafially fdggers 
a poi'ey outcome under a statute. Exampfes inciude wneenera plant or animai 
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when the final panel was announced. While some 
agencies use some of these techniques some of the 
time today, greater transparency needs to become 
the norm, and the processes for assembling advisory 
committees need to become more standardized. 
Merely announcing committees in the Federal Register \s 
not sufficient. 

In addition, each time an advisory committee is 
appointed, an agency should publicly release the 
names of the key officials responsible for appointing 
the advisory committee and briefly describe the roles 
they played in the process. 

Factors in selecting advisory 
committee members 

The primary purpose in appointing a committee is 
to gather a group of eminently qualified individuals 
who can have an open, engaged and comprehensive 
discussion of the issues before them. As the National 
Academy of Sciences puts it in describing its own policy 
for selecting panels, “All [committee members] must 
be highly qualified in terms of knowledge, training and 
experionce. . .to properly address the tasks assigned to 
the committee."* 

Appointing a committee capable of comprehensive 
discussion involves, among otherthings, achieving 
balance among the applicable scientific disciplines. 

Ttiis is more essential than is commonly understood, 
Such balance not oniy ensures that the full range 
of science will inform a decision, but also guards 
against advice being unconsciously biased by the 
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perspectives, values, norms or techniques that may be 
inherent in particular fields. 

Agencies should also strive to include on committees 
scientists at different points in their careers. Young, 
mid-career and senior scientists may have different 
perspectives and strengths to bring to a committee. 

It is also critical to identify a chair who is widely respected, 
has a reputation for considering all perspectives, and can 
manage a committee so as to encourage debate and 
discussion yet produce results on schedule. 

Moreover, agencies should avoid turning 
repeatedly to the same scientists for service on 
advisory committees, instead, they should seek to 
continually expand the circle of relevant experts to 
whom they turn for advice. One way to do this is to 
think more broadly about what expertise is applicable 
to the question before the advisory committee. 
Someone whose work is focused on questions at the 
periphery of a field or subfietd, for example, may have 
sufficient expertise to understand a question and could 
bring a fresh perspective to it. 

Agencies should be alert to their own biases in 
selecting advisory committee members. For example, 
staff who work on an Issue to be reviewed by an 
advisory committee should not select the members of 
that committee, although they should be permitted 
to recommend panelists' names to the official making 
the committee appointments. And agencies should 
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periodically turn over the staff that is assigned to 
select panelists. 

Finally, agencies need To consider what conflicts of 
interest or biases potential committee members may 
bring to the table. To determine whether conflicts or 
biases (as defined below) exist and have been properly 
handled, both agencies and the public need to have 
more information than is currently available concerning 
the members of scientific advisory committees, 

Disclosure of qualifications, 
finances and activities 

Members of federal scientific advisory committees 
should be required to disclose to the government 
information on relevant financial relationships 
and professionai activities (such as giving talks at 
conferences and testifying in court) going back five 
years. Members should also be asked to disclose, to 
the best of their ability, any relevant professional 
activities that occurred more than five years prior 
to their committee service. Any reporting period 
is inherently arbitrary, but the current disclosure 
periods need to be extended to get a fuller picture of a 
member's experience and possible conflicts and biases, 

Federal agency disclosure forms should be as clear 
and uniform as possible; panelists should have no 
doubts about what the government needs to know. 
Developing a single form that draws on the different 
forms used now by the Environmental Protection 
Agency (EPA)* and the Food and Drug Administration 
(FDA)' would be a good start. For example, the FDA 
form asks about speaking and writing, while the 
EPAform does not. The EPAform includes general 
questions on bias that are not on the FDA form. The 
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EPA form asks solely about compensated testimony, 
but uncompensated testimony is also relevant in 
assessing bias; the FDA form is unclear about whether 
uncompensated testimony should be listed.^ 

For finandal disclosure, the categories of 
information the National Academy of Sciences views 
as relevant for its panelists are also appropriate 
forfederat agencies: "employment relationships 
(including private and public sector employment and 
self-employment); consulting relationships (including 
commercial and professional consulting and service 
arrangements, scientific and technical advisory board 
memberships and serving as an expert witness in 
litigation): stocks, bonds and other financial instruments 
and investments including partnerships; real estate 
investments: patents, copyrights and other intellectual 
property interests; commercial business ownership and 
investment interests; services provided in exchange 
for honorariums and travel expense reimbursements; 
research funding and other forms of research 
support, "'’Also, like the Academy, financial disclosure 
should cover not only the irtdividual committee 
member, but "the individual’s spouse and minor 
chiidre.n, the individual's employer, the individual's 
business partners, and others with whom the individual 
has substantial common financial interests... and the 
interests of those for whom one is acting in a fiduciary 
or similar capacity."'" As noted above, the government 
should require this information for the previous five 
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years. The Academy seeks only current information 
about finances.” 

Not only the government, but also the public needs 
more information to determine whether a conflict 
or bias exists and has been appropriately handled. 

To build public trust through transparency, much 
more information on federal advisory committee 
members needs to be available than is now the case. 


To build public trust through 
transparency, much more 
information on federal advisory 
committee members needs to be 
available than is now the case. 


Obviously, a balance must be struck between the value 
of public information and privacy concerns. And public 
disclosure must not be so extensive that it greatly 
reduces the number of scientists willing to serve on 
committees. Federal agencies should monitor whether 
new requirements are making it harder to attract 
committee members. But disclosure is becoming more 
routine - in scientific journals and at universities, for 
example - and the government should not be a last 
bastion of secrecy. 

One possibility would be for federal agencies to 
make publicly available all the information on a 
panelist's disclosure form except the precise dollar 
amounts of their stock holdings or compensation 
and any information on the finances of their spouse 
or dependent children. At the same time, the 
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agency would disclose the member's educational 
background and scientific credentials, ideally, ail of 

this information would be released when committee 
members' names were put up on the Web for public 
comment (per our recommendation above). 

An eventual goal would be to make it standard practice 
for scientists to have a public curriculum vitae (CV) 
that included all their relevant employment, research 
support, publications, speaking, testimony, etc. Such a 
CV would provide much of the information sought on 
government disclosure forms. Many scientists already 
post their CV on their websites, and standardizing and 
expanding this practice would be part of creating a 
culture of disclosure that would be responsive to, and 
relevant for more than requirements for service on 
government committees, 

Regardless of whether they have such a CV, scientists 
should be far more attentive to the need to disclose 
financial relationships and professional activities, 
including the need to disclose any that develop 
during service on an advisory committee. But federal 
agencies must also do their own research on potential 
committee members: they should not rely exclusively 
on self-disclosure. 

Determining conflicts of interest 

While it is the duty of a scientist to fully disclose the 
information needed to determine whether a conflict 
of interest or bias would impinge on service on an 
advisory committee, it is the government that must 
define conflict and bias and dccidehow they will be 
handled. (Bias is dealt with later in this chapter.) For 
conflict of interest, there must be a dearer policy 
with bright lines that leaves as little doubt as 
possible as to who would be considered to have 
a conflict if they served on a particular advisory 
committee. The definition should be as uniform across 


agencies as possible and, at the veiy least, should 
set a minimum standard for ail agencies. Differences 
among agencies lead, among other things, to public 
confusion, and can leave advisory committee members 
open to the charge that a different agency would have 
considered them to have had a conflict of interest. 
Differences among agencies may be acceptable if 
the agencies draw on different scientific fields with 
different norms for conflict, but in such cases agencies 
should be required to explain publicly any departures 
from the standard government definition. (As discussed 
below, defining a conflict Is a separate matter from - if 
related to - deciding what to do when someone is 
determined to have a conflict.) 

The general principles that the National Academy 
of Sciences uses to define conflict of interest apply 
equally well to the government; The term 'conflict 
of interest' means anyfinanciat or other interest 
which conflicts with the service of the individual 
because it (1) could significantly impair the individual's 
objectivity or (2) could create an unfair competitive 
advantage for any person or organization , , ..[Conflict] 
means something more than individual bias. There 
must be an interest, ordinarily financial, that could 
be directly affected by the work of the committee. 
Conflict of interest requirements are objective and 
prophylactic. They are not an assessment of one's 
actual behavior or character. , , .[For regulatory 
issues], the focus of the regulatory inquiry is on the 
identiftcation...of any interests that may be directly 
affected by the use of such reports in the regulatoiy 
process.’’’^ (Italics in the original.) 

Our panel did not reach agreement on a complete set of 
circumstances that should be considered to constitute a 
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conflict of interest. This again underscores the need for 
clear dehnitions and illustrative cases in federal policy 
as the definition is not obvious. (See the Appendices 
of this chapter for a list of the circumstances our pane! 
considered, and for a comparison ofeenflia policies 
used by a variety of institutions.) 

Our pane! did agree that certain relationships should 
be considered a conflict. For example, an employee 
of a company that has a product under review, or a 
scientist funded by that company to research or defend 
that particular product should be considered to have 
a conflict of interest vis-S-vis an advisory committee 
reviewing the environmental or health impacts of that 
product. The same would be true of someone with the 
same links to a compering product. 

The panel also agreed that the question to be asked 
in defining a conflict of interest is whether a particular 
financial relationship would tend to constrain a generic 
individual's point of view, Such relationships need to 
be defined as conflicts regardless of the source of the 
funding. Definitions of conflict should not single out 
scientists based on their affiliation or funding source (Le. 
industry, academia, government, non-governmental 
organisation); rather, conflict policies should treat ail 
paid work in an even-handed manner, that is, according 
to the same principles. An example would be a conflict 
policy That was defining situations in which a scientist 
could fear losing a job or funding if he or she reached 
a particular conclusion. In that case, the task in setting 
policy would be to examine whether each type of 
employment or funding could be construed to pose 
such a threat. In short, it is the relationship between 
the funding source and the scientist, not the funding 
source itself, that is critical. 

Whatever the definition of conflict, when considering 
whether a conflict of interest exists, federal agencies 
should look back two years rather than just 
considering current relationships as Is now the case. 


Any time period Is going to be arbitrary, but financial 
relationships in the immediate past can be a relevant 
consideration. Two members of our panel dissent from 
this recommendation forreasoris explained below.'- 

Agencies should ensure that the theory they use to 
classify particular relationships as conflicts of interest 
{e.g., because such relationships could lead a pane! 
member to fear losing iiiture funding) is consistent 
with the notion of considering past, and not just 
current relationships. 

Dealing with conflicts of interest 

The desired norm forfederal agencies should be to 
appoint advisory committees whose members are 
free of conflicts of Interest. {Relevant experts who 
have conflicts could still make presentations to a panel.) 
There will be instances, though, when scientists with 
conflicts of interest maybe needed for a panel because 
of their expertise. This may be especially true in novel 
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areas of technology in which most funding may come 
from those interested in producing new products. 

Currently, experts w-ith conflicts of interest can be 
appointed as Special Government Employees to serve 
on an advisory committee if the need for their services 
“outweighs the potential for a conflict of interest posed 
by the financial interest involved.*''* The standard 
should be changed to the clearer and arguably 
more stringent policy of the National Academy of 
Sciences under which a conflicted expert can serve 
only in a situation where having a conflicted panel 
member is "unavoidable." The Academy considers 
a conflict to be unavoidable *if, for example, the 
individual's qualifications, knowledge and experience 
are particularly valuable to the work of the committee 
and if the [Academy] is unable to identify another 
individual with comparable qualifications, knowledge 
and experience who does not also have a conflict 
of interest."'^ The Academy’s description of how to 
determine when a waiver is permissible, or one similar 
to it, could be adopted by the federal government even 
if the current statutory language remained unchanged. 

Appointment of an individual with a conflict of interest 
should require a format waiver from the appointing 
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official. When a waiver is granted, the agency should 
publicly state that the appointee has a conflict and 
should provide enough information that the public 
and the other committee members understand 
what kinds of efforts were made to find a non- 
conflict ed individual,'^ how and why the appointed 
individual was considered to be conflicted, and why 
the individual was appointed nonetheless, as well 
as disclosing who signed off on the waiver. (If the 
disclosure procedures proposed in this chapter were 
in place, the agency would still need to specify which 
aspect of the individual's background was considered a 
conflict.) If proposed advisory committee membership 
were placed on the Web for public comment, as 
recommended earlier, that would be the point at which 
a waiver would be announced. 

Agencies should not appoint anyone with a conflict 
to serve as the chair or co-chair of a committee. And 

agencies should limit the issuing of conflict waivers to 
ensure that individuals with conflicts do not generally 
constitute more than a small percentage of the 
membership of a committee. 

Determining and dealing with bias 

The federal government should follow the National 
Academy's lead and distinguish clearly between 
conflict of interest and bias. The Academy's view of 
bias should guide federal policymakers: "Questions of 
lack of objectivity and bias ordinarily relate to views 
stated or positions taken that are largely intellectually 
motivated or that arise from the close identification 
or association of an individual with a particular point 
of view or the positions or perspectives of a particular 
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group. Potential sources of bias are not necessarily 
disqualifying for purposes of committee senrice."’' 

Bias is an inherently murky concept, and every 
individual has biases. But the goal should generally be 
to assemble committees of individuals who are as 
impartial (i.e., fair-minded) as possible. This is not the 

same as saying that a committee should (or could) be 
made up of individuals with no views on the matter at 
hand; the goal is to pull together a committee that can 
act in good faith. 


Agencies should not shy away 
from including scientists on 
a panel who arc considered 
“outliers” on the question(s) 
under consideration, provided 
that the scientist is a respected 
practitioner in a relevant field and 
the coinmittce as a whole fairly 
represents the mainstream. 


The approach to bias wii) depend on the precise 
question(s) being posed to the committee. Generally, 
strong biases in committee members should be 
avoided, But in some cases, an agency may want 
to appoint some members with strong and even 
fixed views on an issue because they need such 
individuals' expertise or because they want to ensure 
that those scientific views are fully represented on 
the committee. In such instances, the goal should be 
to ensure that the overall committee is balanced. 
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Agencies should not shy away from including 
scientists on a panel who are considered "outliers" 
on the question(s) under consideration, provided 
that the scientist is a respected practitioner in a 
relevant field and the committee as a whole fairly 
represents the mainstream. Minority reports should 
be allowed on advisory committees, although 
consensus should be the goal.’^ Outliers who are 
willing to engage the issues can play a useful function 
by sharpening discussion within a committee, even 
if they decide not to sign on to the committee's final 
report. 

Unlike conflict of interest, there is no way to come 
up with a litmus test for bias or to establish clearly 
delineated categories. Rather, in handling bias, federal 
agencies need to carefully consider the full picture 
of an individual's activities that emerges from his 
or her disclosure forms as well as getting a sense 
of the individual's personality and reputation in the 
field. For example, for academic scientists, receiving 
funding from a variety of sources can be a sign of fair- 
mindedness, Similarly, responding to critics, publishing 
in a variety of journals, and speaking at a variety of 
invited conferences can be indicators of openness, On 
the other hand, testifying repeatedly on one side of 
an issue before Congress or in the courts can be taken 
as indications of a point of view that may need to be 
balanced in putting together a panel. 

Managing advisory committees 

Once the final members of a committee have been 

named, federal agencies need to defend their 
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choices of appointees and stand by their panel if it 
comes under attack, (This assumes, of course, that 
no new information comes to light that should have 
been disclosed by a scientist or uncovered by an 
agency. Seeking public comment on committee 
members, as recommended earlier, should help 
prevent such situations.) 

Committee members need to know of each other's 
financial relationships and viewpoints. At their first 
meeting, and periodicallythereafter,Nationa! Academy 
panel members are expected to discuss their relationships 
and previously stated views with fellow panel members 
in a closed session, Federal agencies may want to adopt 
this practice (although under current law it would have 
to be in open session), or may want to experiment with 
other means of ensuring that a panel has a coileaive 
understanding of Its membership’s commitments and 
interests. At a minimum, advisory committee members 
should be given copies of ai! the members' public 
disclosure forms prior to the first meeting. 

Moreover, the appropriate agency official needs to 
take an active role in supporting the committee's 
work, which includes managing any conflicts for the 
duration of the panel. An official may need to remind 
pane! members of member interests. Also, an official 
may need to seek recusai of a member or otherwise 
manage conflicts, if they develop.’^ New conflicts that 
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develop or relevant new activities that are undertaken 
during service on a committee must be disclosed and 
handled in the same manner they would have been in 
advance of service on a panei. 

Federal agencies should not be able to circumvent 
the processes discussed above by contracting 
out the appointment or operation of advisory 
committees. The Administration should limit the 
extent to which federal agencies can use outside 
contractors to establish advisory conimittees, and 
federal agencies should be aleit to any conflicts of 
interest those firms may pose. Moreover, committees 
chosen by contractors should be subject to the same 
ruies and procedures as a similar committee established 
directly by an agency, particularly on the matters of 
conflict, bias and disclosure discussed above, 

It is also vital for the federal government to establish 
and maintain an internal tracking system on the process 
of recruiting scientific advisors, the numbers and types 
of conflicts and biases encountered and the degree to 
which increased disclosure Inhibits the recruitment of 
a full range of qualified experts. In addition, the public 
database on advisory committees needs significant 
improvement. It should provide easy access to the 
names and backgrounds of ail individuals serving on 
advisory committees and information on the conflict of 
interest waivers chat have been granted. 

The Administration and the Congress should carefully 
think through the benefits and disadvantages of requiring 
ail meetings to be open. It might be worth considering, 
for example, whether some scientific advisory 
committees could be allowed to hold some closed 
meetings if the selection process for committee 
membership were more open than it generally is 
today (as recommended above). Transparency is an 
essential principle of democratic governance, but some 
deliberations can benefit from a modicum of private 
discussion to enable committee members to think and 
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speak more freely and open-mindedty. Allowing the 
closure of meetings would require changes in statute, and 
any such changes should limit the use of closed meetings 
and be very specific about when closure is permissible. 

The recommendations of a committee, though, 
must always be made public (assuming no 
classified information is involved), and indeed 
committees should be required to explain fully their 
methodology and the rationale for their conclusions, 
in the Federal Register notice for any rule for which 
a scientific advisory committee was convened, the 
federal agency should be required to state whether 
it differed with any conclusions of a scientific 


advisory committee and if so, why, and should be 
required to explain how the new regulatory policy is 
consistent with the conclusions that were accepted. 

Finally, federal officials must give advisory committees 
clear, definite and realistic deadlines for reporting and 
clear information on when a committee report will be 
released and how it will be used. 

One way the Administration might approach some of 
the issues raised here would be to review the guidance 
that the Office of Management and Budget and the 
Office of Science and lechnology Policy issued in 2005 
to see how it might be improved.^® 
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APPENDIX 1 


Hypotheticals for consideration in setting rules for conflict of interest 


As noted in the text, our panel did not reach agreement 
on a complete set of circumstances that should be 
considered to constitute a conflict of interest. But we did 
have a detailed discussion about the circumstances that 
might constitute a conflict. To structure that discussion, 
we debated the hypothetical cases described below. 
(Some of the hypotheticals are based on actual cases.) 

Our panel did not agree on whether to define these 
cases as examples of conflict of interest or bias, or on 
whether to exclude the individual described in the case. 
Notably, though, these were two separate questions. 

For example, there were a number of members of our 
group who would describe these cases as "bias" but 
would nonetheless generally exclude the person with 
the bias (rather than just balancing their presence). The 
cases are described below because they should be 
thought through by any official deciding how to define 
and handle conflict and bias. 

■ An individual is a board member, employee or 
significant stock holder of the company whose 
product is being reviewed by an advisory committee 
- or has a similar interest in a competing company. 
How should company be defined for these cases? 
Would the limitation be the same if the byproduct of 
a company's production was being reviewed, esp. if 
that byproduct was produced by many companies 
or even industries? 

■ An individual has received funding from the 
company to study the particular product 
under review. 

■ An individual has received funding to study the 
particular product under review from a philanthropic 
entity set up by the company (and that maintains 
close ties with the company). 


• An individual has received research funding from a 
company that has a direct interest in the results of an 
advisory committee, but on a different subject - 
maybe even from a differeru division of the 
company. Should that person be excluded? Should 
it depend on whether the individual has also 
received funding from the government or others? 
Should it depend on whether the individual’s work 
has generally or always supported the company's 
point of view? 

■ An individual is a board member, employee or 
stockholder of a company that would be part of a 
general class of companies affected by a regulation - 
say, a clean air rule. Should the disqualification 

still be automatic? Should it matter what division 
of a company the person is associated with (if 
an employee)? 

« An individual is an employee of a non-governmental 
organization (NGO) that has taken a position on the 
issue before the committee. 

■ An individual is president of a professional society 
that espouses a position on the issue that is 
under review. 

■ An individual is an unpaid board member of an NGO 
that has taken a position on the issue before the 
committee. What if the individual is a board member 
ofan environmental group on an issue on which 
other environmental groups have weighed in? 

■ An individual is an employee ofan NGO and that 
individual has publicly testified on the matter 
under review by the committee. Would it make a 
difference if that testimony occurred during service 
on the committee? 
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■ An individual runs a university center that has 
received funding from a company with a direct 
interest in the matter under review. Would the same 
decision apply to someone who was a dean of a 
college or president of a university that received that 
funding? Would the same decision apply to a gift 
from an individual with a clearly held view -either 
on the specific issue or ideologically? [Do such items 
even require disclosure?] 

■ An individual is affiliated with (but does not head) a 
university center that has received funding from a 
company with a direct interest in the matter under 
review. What if the person were a professor in a 
college or university that received such funding? 

■ An individual received funding from a federal agency 
that has an interest in the outcome of a review (as a 
regulated entity). 


■ An individual has consulted for a company that has 
a direct interest in the matter under review. How 
dose does the consultation have to be related to the 
matter at hand? Does it matter if the consultation 
was arranged through a contractor that was helping 
to defend a company's product? 

■ An individual has a consulting contract with a firm 
whose other clients include a company with a 
matter under review by an advisory committee. 

■ An individual was paid for lectures by a company 
with a product under review. Does it matter how 
directly the lectures promoted the product? 

■ An individual is a member of a political or policy 
advocacy group that has taken a strong stand on the 
issue in question. 
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Chapter Three 


RECOMMENDATION THREE: Agencies and their scientific advisory 
committees should cast a wide net in reviewing studies relevant to regulatory 
policy, and should make their methods for filtering and evaluating tliose 
studies more transparent. 


it is a commonpiace to (irgue that regulation should be 
based on the "best available science,” but determining 
what constitutes the best available science in any 
specific instance is no easy task. Assembling and 
evaluating the relevant scientific literature is a complex 
undertaking, not subject to any single, simple formula. 
That said, some basic principles should guide agencies 
and their scientific advisory committees as they sift the 
scientific literature, it should be Administration policy 
that agencies adhere to these principles. 

Transparency 

First, the process of conducting literature reviews 
should become more transparent. Agencies and 
their scientific advisory committees should be explicit 
about the criteria they are using to determine which 
scientific papers to review and how those papers are 
being evaluated. Those criteria should be open for 
public comment either as part of the comment period 
on a proposed rule or, when possible, earlier in the 
rulemaking process. 

In addition, once an agency has opened a docket 
on a rule or guidance that will draw on scientific 
studies, it should make available on the Web a list 
of the studies it is reviewing and should regularly 


update the list’ The list should be open for public 
comment both to help evaluate the studies on the 
list and to help identify any relevant studies that are 
being omitted. When a rule based on scientific studies 
is proposed, agencies should make clear in the Federal 
Register notice which studies were particularly influential 
and why.^ Agencies should require their scientific 
advisory committees to do the same in their final reports. 

Criteria 

While the specific criteria an agency or scientific 
advisory committee uses to evaluate scientific studies 
may vary from issue to issue, the criteria should always 
be consistent with the principles below. 

In general, papers in high impact, peer reviewed 
journals should be given great weight, and papers 
that have not been peer reviewed should be 
treated with skepticism. However, the quality of 
peer review varies widely, and journal rankings and 
impact factors do not guarantee that peer review of 
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a specific paper was performed adequately. Agencies 
and scientific advisory committees need to extend 
their inquiry beyond simply ascertaining whether 
a paper was peer reviewed; peer review Is a 
necessary but not sufficient determinant of quality. 
That further inquiry might explore how the peer review 
was conducted, how the paper fits into the larger body 
of literature under review, and perhaps most important, 
the methodology behind the conclusions described 
in the paper (for example, how a cohort to study was 
chosen in an epidemiological study). 


In general, agencies and scientific 
advisory committees should be 
wary of studies when it is unclear 
who funded the study or whether 
the principal inve,stigator(s) had 
any conflicts of interest. 


Conversely, studies that have not been peer reviewed 
should not be summarily rejected if they appear to 
contribute to the inquiry. Agencies and scientific 
advisory committees should be able to commission 
their own peer reviews (or in some cases, the scientific 
advisory committee itself might be assigned, or take on 
peer review of a study as a forma! task). Agencies need 
sufficient funding and need to set realistic schedules to 
ailow for such reviews. 

(Chapter 4 discusses ways in which the federal 
government, as well as scientists and scientific Journals 
could improve the peer review process.) 

In general, agencies and scientific advisory 
committees should be wary of studies when 
it is unclear who funded the study or whether 
the principal investigatoris) had any conflicts of 


interest.^ Agencies and scientific advisory committees 
can seek this information if it is not made public as part 
of the paper itself.^ Agencies and scientific advisory 
committees should consider sources of funding and 
any conflicts of interest as they review the reasons why 
a study may have been undertaken, the way a study 
was framed and carried out, and how the study results 
have been interpreted and discussed, in general, no 
studies should be excluded o priori because of the type 
of funding behind them.^ The focus should be on the 
study itself. 

Beyond general concerns about funding and conflicts 

of interest, agencies and scientific advisory 
committees should be extremely skeptical of 
a scientific study unless they are sure that the 
principal investigator(s} (as opposed to the sponsor 
or funder) had ultimate control over the design and 
publication of the study. 

In fields where a public registry of studies exists 
(such as the registry established by the Food and 
Drug Administration Modernization Act of 1997), 
agencies and scientific advisory committees should 
consider the relevant registered studies and should 
be wary of studies that met the criteria for the 
registi 7 , but were not registered. Among the reasons 
to consult a registry is that a registry is more likely than 
the published literature to include reports of negative 
results (i.e., of instances where a study failed to confirm 
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an expected effect). Negative results need to be taken 
into account even when they are not peer reviewed 
(with the cautions mentioned above); they are less 
likely to be peer reviewed because journals are often 
reluctant to publish such studies. 

There is no simple way to lay out generic rules for 
literature reviews - every body of literature and every 
field has its idiosyncrasies •• but the principles above 
should offer some overarching guidance. In short, a 
good literature review strives to develop a sense of the 
entire body of relevant literature; evaluates the methods 
that were used in studies: digs, when necessary, beyond 
the published material, to get a better sense of methods 
and data; and is aware of the sources of funding and the 
extent of sponsor control over studies. Or, put another 
way, a good literature review is an exercise in comparing 
studies, looking first at the thrust of a body of literature 
and how broadly and well founded its conclusions are, 
then examining any well done studies that may be 
taking issue with the literature, and then reviewing what 
might be categorized as exploratory studies - studies 
that may relate to the question under consideration but 
were not carried out for that purpose. 

Data availability 

As noted above, literature reviews are enhanced when 
more information is available on the methods and data 
on which studies' conclusions are based. Scientists 
themselves and scientific journals could take steps 
to facilitate access to methods and data, as will be 
discussed in the next chapter. But the government 
also could increase the availability of information on 
methods and data.* 

Studies used in the formulation of regulation 
should be subject to data access requirements 


O'^r wnci n'or:’'<i:ionQL'3rtyActff’I-106--5S4.S«cLioo515)or'he 


equivalent to those under the Data Access Act 
(Shelby Amendment)^ and its implementing 
circular^ regardless of who funded the study. If a 

study is used by an agency to inform the development 
of a regulation, then the same kinds of information 
about that study should be available upon request, 
regardless of whether the study was funded by the 
federal government, industry, or some other entity. 

Confidential Business Information (CBI) claims can 
also make it difficult for the interested public to 
evaluate studies that contribute to regulatory policy, 

CBi is a legitimate and needed designation for 
information submitted to the federal government, 
but it appears to be overused today. There is great 
incentive for companies to claim CBI (i.e., why not 
err on the side of caution and secrecy?) even though 
that may be counter-balanced by a desire to earn the 
trust of regulators by being open about their scientific 
data. The Administration and the Congress should 
gather data on the extent and nature of CBI claims. 
The Administration and Congress should consider 
requiring each new CBi claim to include a brief, 
but substantive justification forthe claim. Congress 
should also review the CBI provisions of specific statutes 
as they come up for reauthorization. 

Additional studies 

The recommendations in this report thus far, though, 
may not always be enough to resolve a regulatory 
question or to gain public faith in a regulator process, 
Sometimes, a literature review will make clear the need 
for additional studies to address a dispute, but an agency 
will be seen as having too great a stake in the outcome 
to commission such work itself. Or sometimes an agency 
may not be trusted to appoint a balanced scientific 


■ °.L. 105-277. 

* OMSCircu'arA-ilO. 



297 


44 


Improving tiic U^e of Science in Regulatory Policy 


advisory committee, or the complexity of doing so might 
strain its resources. In the latter case, an agency could 
turn to the National Academy of Sciences, but agencies 
may want to consider creating other avenues, especially 
if settling a controversy requires additional studies, not 
just a review of the existing literature. 

!n the area of clean air policy, the Health Effeas Institute 
(HEl), established in 1980 and jointly funded by industry 
and government, has established a reputation as an 
honest broker with trusted scientific expenise to help 
the Environmental Protection Agency when it runs 
into the kinds of issues mentioned in the previous 
paragraph. HE! has clear and strict procedures for 
commissioning and reviewing studies that have 
enabled it to be seen by aii sides in clean air disputes as 
an unbiased authority.'-’ 

Agencies should experiment with a variety of 
additional approaches that would enable them 
to commission studies and literature reviews 
related to pending regulatory decisions that 
would be widely seen as unbiased. For example, 
agencies might want to consider setting up their own 
equivalents of HEl, or they might want to consider 
giving either standing or ad hoc scientific advisory 
committees the ability (and the budget) to commission 
additional studies. They might also turn to another 
federal entity that would not be considered to have a 
stake in the outcome of the issue. Regulatory agencies 
have sometimes turned to the National institute of 
Environmental Health Sciences for this purpose, for 
example. 
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Agencies should also encourage creative mechanisms 
by which scientists from industry, government, academia 
and non-governmental organizations can design 
experiments, coliaborate on studies, and co-author 
scientific papers for pubiication in the open literature. In 
addition to advancing scientific knowledge, these multi- 
sector collaborations may work to buiid trust. 

Presenting conclusions 

In presenting the conclusions of literature reviews, 
agencies and their scientific advisory committees 
need to be as open and precise as possible in 
discussing levels of risk and uncertainty. Deciding 
how much risk and what kinds of risk society should 
tolerate is a policy decision, as is determining whether 
and how to act in the face of scientific uncertainly, 
Those values questions need to be debated fully and 
openly. What agency scientists and scientific advisory 
committees need to do to inform that debate is provide 
clear scientific information about what the risks appear 
to be and how definitive the current scientific literature 
is about the existence and levels of those risks. 

Policy makers should be wary of conclusions about 
risk that are expressed as a single number. Rather, 
risk should be expressed as a range, with different 
scenarios and assumptions for different risk levels, 
including their relative likelihoods, spelled out. The 
population distribution of risks should be speiied out 
when such information is available and relevant. Also, 
terms that are applied to levels of risk (e.g,, "probable" 
or ’possible") need to be defined precisely, i.e. 
quantitatively. Legal terms need to be translated into 
scientific ranges and vice-versa. The same is true for the 
terminology used to describe uncertainty. 

if agency scientists or a scientific advisory panel concludes 
that a range of concentrations is safe for humans, animals 
Of plants, it should be dear about the levels of uncertainty 
and risk at different levels within that range. 
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RECOMMENDATION FOUR: The federal government, universities, 
scientific journals and scientists themselves can help improve the use of 
science in the regulatory process by strengthening peer review, expanding the 
information available about scientific studies, and setting and enforcing clear 
standards governing conflict of interest. 


This report has focused on steps the federal 
government needs to take to clarify and improve 
its own processes for injecting science into the 
regulatory process. But the federal regulatory process 
draws on the larger ecology (and economy) of the 
scientific enterprise and can only improve to the 
extent that the overall enterprise is functioning well. 

It will be more difficult for the federal government 
to achieve the improvements called for in our earlier 
chapters unless the other actors in the scientific 
enterprise also rise to the occasion. 

Peer review 

Peer review is the primary guarantor of integrity in 
the scientific system. It has inherent limitations, as 
do all human processes, but without it, the scientific 
enterprise would have diminished quality and 
credibility, in recent years, there has been growing 
concern that the peer review system may be eroding. 
Scientists may feel too burdened to review their 
colleagues’ papers or may do so with insufficient care. 
Peer review is no longer assumed to be a professional 
obligation, and the institutions that rely on peer review 
mostly do too little to underscore its value. Moreover, 


there has been little experimentation or empirical study 
about how to make it more effective. 

Federal agencies need to experiment with ways to 
inaease the number of scientists who participate 
in peer review, particularly peer review of draft 
manuscripts. Possible steps that could be tried run 
the gamut from paying scientific advisory committee 
members a nominal fee to participate to requiring federal 
grantees to participate in a minimum number of peer 
reviews over the life of their grant to qualify for future 
funding. A middle-ground might be requiring grant 
applicants to list peer review service on their applications. 
The government could also encourage or require 
universities that receive federal grants to demonstrate 
that they were creating incentives for their faculty to 
participate as peer reviewers. It might help even just 
to have top federal science officials make clear in their 
speeches and writings that ser\'ice as a peer reviewer 
is an expected aspect of a scientific career. Agencies 
should also ensure that their own scientists serve as peer 
reviewers. The Office of Science and Technology Policy 
could dirert federal agencies to experiment with ways to 
increase participation in peer review, and then evaluate 
which experiments turn out to be most successful. 
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Universities should do more to make service as a 
peer reviewer an expected and appreciated aspect 
of a scientist's career. Faculty should be encouraged 
to participate in peer review by both their colleagues 
and administrators. Service as a peer reviewer should 


Peer review is die primary 
guarantor of integrity in the 
scientific system. 


be rewarded in tenure, promotion and salary decisions, 
Graduate students, post-doctora! researchers, and even 
faculty, particularly juniorfaculty, should be mentored 
on how to conduct a creditable peer review. 

Scientific journals should Improve the quality 
control of peer review and should experiment with 
different ways of conducting peer reviews. Journals 
should try to expand their circle of peer reviewers and 
should encourage more thorough peer reviews, perhaps 
by publicly acknowledging top-notch peer reviewers. 
Journals should give peer reviewers feedback {perhaps 
from the scientists whose work they reviewed) on the 
quality of their peer reviews. Journals should consider 
experiments to determine what produces the best (and 
in some cases, most transparent) peer reviews, such 
as publishing peer reviews (with or without the name 
of the peer reviewer) on the Web or along with the 
paper being reviewed, publishing lists of peer reviewers 
annually or in each issue, using open rather than 
anonymous peer reviews, or going in the other direction 
and using double-blind peer reviews. Journals should 
require their peer reviewers to disclose to the Journal 
the information the editors need to determine whether 
any conflicts of interest exist, and thejourna! should 
consider disclosing that information to the article author 


and/or the readership.' Also, journals could consider 
nominally compensating reviewers. 

Information on scientific studies 

As noted in Chapter 3, to evaluate a study fairly 
and completely, one needs a full sense of the data 
generated and the methods employed in the study. Yet 
this information is often difficult to obtain. That may 
have been understandable when paper journals were 
the basic means to communicate scientific information, 
but in the electronic age, the primary limitation on 
providing more information is a lack of will. Federal 
agencies, universities and journals should 
encourage or require on-line publication of the 
methods and data underlying published scientific 
studies. The extent to which data and methods 
should be made pubfe will vary by field, as each 
field has different standards as to what information a 
scientists can hold close to protect their intellectual 
property or future work. But enough information 
should be published on-line in conjunction with journal 
publication that an interested scientist could fairly 
evaluate the study results and replicate them, if so 
desired- Scientists need to understand that if they wish 
their studies to be retied upon by federal regulators, 
those studies must have a high degree of transparency 
about data and analytic methods, 

Also as noted in Chapter 3. registries can help make 
information on a field of research more complete 
and accessible. Federal agencies should determine 
whether the idea of research registries, which today 
is focused on research related to pharmaceuticals, 
can be expanded to other fields. The nature of 
clinical trial research and its link to federal regulation 
may well be unique, but there may be other fields 
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with enough similarities to experiment with variations 
on registries. At the very least, agencies could make it 
easier to find the results of any study that was federally 
funded by, for example, having a searchable database 
of reports on what research was performed with grant 
money, A model might be the EPA STAR (Science 
to Achieve Results) grant program, which maintains 
on-line versions of the regularly submitted progress 
reports from projects it is funding, providing a broader 
inventory of the work underway than is available 
through the peer reviewed literature. Ideally, such a 
database would also include intramural research, that 
is, research that was carried out by federal agencies 
themselves rather than by grantees. 

Conflict of interest 

As noted in Chapter 3, the quality of literature reviews 
depends, in part, on having complete and accurate 
information on the funding of scientific studies and 
on any possible conflicts of interest that the scientists 
conducting a study may have had, Given the value 
placed on peer review and on the reputations of 
scientific journals, it is also vital that any conflicts editors 
may have be disclosed as welt. Many journals have 
tightened their conflict of interest rules in recent years, 
and they have made some efforts to coordinate their 
policies. But more could be done. Journals should 
have clear, publicly accessible conflict-of-interest 
policies and should require full disclosure of how 
studies were funded and of any and all conflicts 
of interest they determine an author has. Editors 
should also disclose any of their own conflicts 
of interest. In addition, journals should consider 
requiring authors to certify that they had ultimate 


control over the design and publication of the 
study being described in a paper. 

As noted in Chapter 2, universities have also begun 
to put in place tighter and more consistent rules 
concerning conflict of interest. Universities need to help 
create a culture of transparency about funding and need 
to have clear, accessible and enforced policies on conflict 
of interest and on ultimate sponsor control. Violating 
university policies should have real consequences. 

Similarly, federal agencies need to consider 
promulgating rules that would sanction scientists 
who run afoul of federal, university or Journal 
requirements concerning disclosure of conflict of 
interest or ultimate sponsor control. 

Scientists 

The recommendations above are directed at institutions 
both to help create a culture of participation and 
transparency and to ensure that bad actors are 
discovered and reprimanded. But a truly healthy 
scientific enterprise relies on the individual actions and 
the decisions of each scientist. Scientists themselves, 
regardless of where they work, need to understand 
that the future health and credibility of the scientific 
enterprise depend on individual scientists addressing 
the concerns raised in this chapter. They need to ensure 
that they and their colleagues are participating as actively 
and openly as possible in the entire scientific proce.ss 
from research through publication and are open to 
involving themselves in the policy process. They should 
work as well to ensure that their professional societies 
regularly host sessions at their annual meetings on the 
importance and conduct of peer review, and consider the 
establishment of annual awards for particularly significant 
contributors to peer review in their fields. Scientists 
cannot expect regulatory policies to be based on the 
best available science unless they conduct, review, and 
evaluate that science in a way that garners public trust. 
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Afterword 


Our report does not, and was not intended to deal with 
every issue that bedevils regulatory policy making, or 
even the use of science in it. Our panel focused on what 
we saw as perhaps the most fundamental and least 
discussed problems in regulatory policy making - the 
conflation of science and policy questions, and the 
need for greater transparency in analyzing the science 
behind policy making, 

Among the many questions we did not discuss, but 
want to acknowledge are: how to strengthen the 
internal scientific capacity of federal agencies, how to 
protect whistleblowers, the extent to which the White 
House (and in particular, the Office of Information and 
Regulatory Affairs) should review specific regulatory 
decisions, and what kind of access individual federal 
scientists should have to the media. These and other 
questions are important, but other groups have 
weighed in on them, and we put these matters beyond 
the purview of our report. 

There are two matters, though, that we want to point 
out that should draw attention from those both inside 
and outside the government who might wish to follow 
up on this report. 

First, as noted in the report, there is remarkably little 
empirical data and relatively little discussion in the 
policy literature of the issues the report covers. Data 
and research are greatly needed on such questions 
as: Who is getting appointed to federal advisory 
committees and how? How many advisory committee 
members have conflicts of interest (however 


defined) and what impact do Those conflicts have on 
committee proceedings? What kinds of committees 
give the best advice? What kinds of literature reviews 
are most "successful"? How often do peer reviewed 
papers prove to have faulty methodologies and how 
can that be prevented? What peer review systems 
work best? And so on. More work is also needed on 
questions related to the central theme of this report 
- the need to distinguish between scientific and 
policy questions. How can that be done in specific 
cases? Where has it been done successfully? What 
is the impact? What other, broader changes to the 
political system might enhance the debate about 
science and regulation? Our panel drew on the 
considerable and varied persona! experience of our 
members and the policy literature that does exist to 
develop recommendations that we believe will make 
a difference. But that old saw of scientific reports 
is especially valid here: more research is needed, in 
this case, the research should include monitoring 
the extent to which our recommendations are 
implemented and their impact. 

Second, this report did not deal directly with one . 
fundamental problem at the intersection of science 
and policy; the inherent disconnect between the pace 
at which scientific understatiding changes and at 
which policy action takes place, Sometimes the policy 
apparatus cannot keep up w'ith the speed of scientific 
change; in other cases, policy makers seek scientific 
answers before the research to provide them is ripe. 
Deeper thinking is needed on the question of how to 
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continually refresh the scientific understanding that 
underlies regulatory policy, and how to periodically 
update that policy as a result - without building in 
so much instability that irtdustries cannot plan, or so 
much constant debate that the rulemaking apparatus 
simply seizes up entirely. 

Science and politics are both dynamic systems, 
and this report will hardly be the last word on the 
intersection of science and regulatory policy. But we 
believe it is an important start. We look forward to 
working with the Administration and the Congress to 
implement our recommendations. 
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on such trans-boundary environmental problems as: 
major land-use changes; economically-driven alterations 
in agricultural practice; global climate change; and the 
development of regulatory policies. Dr. Kennedy has 
served on the faculty of Stanford University since 1960, 
From 1980 to 1992 he served as President of Stanford 
University. He was Commissioner of the U.S. Food 
and Drug Administration from 1977-79. Previously at 
Stanford, be was Director of the Program in Human 
Biology from 1973-77and Chair of the Department 
of Biology from 1964-72. Kennedy is a member of the 
National Academy of Sciences, the American Academy 
of Arts and Sciences, and the American Philosophical 
Society. He served on the National Commission for 
Public Service and the Carnegie Commission on Science, 
Technology and Government, and as a founding Director 
of the Health Effects Institute. He currently serves as a 
Director of the Carnegie Endowment for international 
Peace, and as Co-chair of the National Academies' Project 
on Science, Technology and Law, 

ARTHUR CAPLAN 

Arthur Caplan is the Emanuel and Robert Hart Professor 
of Bioethics, Chair of the Department of Medical Ethics 
and the Director ofthe Center for Bioethics at the 
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University of Pennsylvania. Prior to coming to Penn in 
1994, Dr, Caplan taught at the University of Minnesota, 
the University of Pittsburgh, and Columbia University. He 
was the Associate Director of the Hastings Center from 
1984-87. Dr. Caplan is the author or editor of 25 books 
and over 500 papers in refereed journals of medicine, 
science, philosophy, bioethics and health policy. His 
most recent book is Smart Mice Not So Smart People 
(Rowman Littlefield, 2006). He has served on many 
national and internationai committees including as the 
Chair of the National Cancer Institute Biobanking Ethics 
Working Group, the Chair of the Advisory Committee to 
the United Nations on Human Cloning, the Chair of the 
Advisory Committee to the Department of Health and 
Human Services on Blood Safety and Availability, and a 
member of the Presidential Advisory Committee on Gulf 
War Illnesses. He is a member of the Board of Directors 
of The Keystone Center, Tengion, the National Center 
for Policy Research on Women and Families, Octagon, 
the Iron Disorders Foundation, and the National Disease 
Research Interchange. He writes a regular column on 
bioethics for MSNBC, com. Dr. Caplan is the recipient 
of many awards and honors including the McGovern 
Medal of the American Medical Writers Association, 
Person of the Year-2001 from USA Today, one of the 
50 most influential people in American health care 
by Modern Health Care magazine, one of the 10 most 
influential people in America in biotechnology by the 
National Journal and one of the ten most influential 
people in the ethics of biotechnology over the past ten 
years by the editors of the journal Nature Biotechnology. 

LINDA J. FISHER 

Linda J, Fisher is Vice President and Chief Sustainability 
Officer at E. I, du Pont de Nemours and Company. 

She has responsibility for advancing DuPont's 
progress in achieving sustainable growth, DuPont’s 
environmental and health programs, the company's 
product stewardship programs, global regulatory 


affairs, and government affairs. She joined DuPont 
in 2004. Prior to that, Fisher served in a number of 
key leadership positions in government and industry 
including: Deputy Adminfstrator of the Environmental 
Protection Agency (EPA) from 2001-03; EPA Assistant 
Administrator - Office of Prevention, Pesticides and 
Toxic Substances; EPA Assistant Administrator - Office 
of Policy, Planning and Evaluation; and Chief of Staff to 
the EPA Administrator. Fisher, an attorney, was also Vice 
President of Government Affairs for Monsanto and was 
Of Counsel with the law firm Latham & Watkins. She is 
a member of the DuPont Health Advisory Board and 
the DuPont Biotechnology Advisory Panel and serves 
as liaison to the Environmental Policy Committee of the 
DuPont Board of Directors. Fisher serves on the Board 
of Directors of the Environmental Law Institute, on the 
Board ofTrustees of TTie National Parks Foundation, on 
the Board of Directors of Resources for the Future, and 
on the Board of Covanta Holdings. 

LYNN R. GOLDMAN 

Lynn R, Goldman, a pediatrician and epidemiologist, is , 
Professor in the Department of Environmental Health 
Sciences at the Johns Hopkins University Bloomberg 
School of Public Health. Her areas of focus are public 
health practice, children's ertvironmental health, 
disaster preparedness, and chemical and pesticide 
regulatory policy. Dr. Goldman is Principal investigator 
for the Hopkins National Children's Study Center and 
co-PI of the Center for Preparedness and Catastrophic 
Event Response (PACER), As Assistant Administrator 
for Toxic Substances at EPA. she directed the Office 
of Prevention, Pesticides and Toxic Substances from 
1 993 through 1 998. Prior to joining EPA, Dr. Goldman 
served as Chief of the Division of Environmental 
and Occupational Disease Control of the California 
Department of Health Services. Dr. Goldman has 
served on numerous boards and expert committees, 
including the Committee on Environmental Health of 
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the American Academy of Pediatrics and the Centers 
for Disease Control Lead Poisoning Prevention Advisory 
Committee. Dr. Goldman is a member of the Institute 
of Medicine, Vice Chairman of the institute of Medicine 
Roundtable on Environmental Health Sciences, and a 
member of the National Academy of Sciences Standing 
Committee on Risk Analysis Issues and Reviews. 

JOHN D. GRAHAM 

John D, Graham Is Dean of the Indiana University 
School of Public and Envirenmentat Affairs (SPEA). His 
research interests include government reform, energy 
and the environment, and the future of the automobile 
in both developed and developing countries. He came 
to SPEA after serving as Deart of the Frederick Pardee 
RAND Graduate School at the RAND Corporation in 
California. Prior to Joining RAND, Dr. Graham served 
in the White House Office of Management and 
Budget {0MB) from 2001-06, As the Senate-confirmed 
Administrator of the Office of Information and 
Regulatory Affairs, he led a staff of 50 career policy 
analysts who reviewed major regulatory proposals from 
Cabinet agencies. Prior to his role at 0MB, Dr. Graham 
was a Professor of Policy and Decision Sciences at the 
Harvard School of Public Health. From 1990 to 2001, 

Dr. Graham founded and led the Harvard Center for 
Risk Analysis, in 1995, he was elected President of the 
Society for Risk Analysis, an international membership 
organization of 2,400 scientists and engineers. 

DANIEL GREENBAUM 

Dan Greenbaum joined the Health Effects institute 
(HEI) as its President and Chief Executive Offeer in 
1 994. In that role, Greenbaum leads HEI's efforts, 
supported jointly by the EPA and industry, with 
additional funding from the Department of Energy, 
Federal Highway Administration, U.S. Agency for 
international Development, the Asian Development 


Bank, and foundations, to provide public and private 
decision makers with high quality, impartial, relevant 
and credible science about the health effects of air 
pollution. Greenbaum has focused HEI’s efforts on 
providing timely and critical research and reanalysis 
on particulate matter, air toxics, diesel exhaust and 
alternative technologies and fuels. Greenbaum 
currently serves on the U.S. National Research Council 
(NRO Committee on Health, Environmental, and Other 
Externa! Costs and Benefits of Energy Production 
and Consumption. He has been a member of the 
NRC Board of Environmental Studies and Toxicology 
and Vice Chair of its Committee for Air Quality 
Management in the United States. Greenbaum also 
chaired the EPA Blue Ribbon Panel on Oxygenates in 
Gasoline, which issued the report "Achieving Clean Air 
and Clean Water' and EPA's Clean Diesel Independent 
Review Panel, which reviewed technology progress 
in implementing the 2007 Highway Diesel Rule. 

Before coming to HEi, he was Commissioner of 
Environmental Protection in Massachusetts. 

MICHAEL P. HOLSAPPLE 

Michael P. Holsapple is the Executive Director of 
the Internationa! Life Sciences institute's Health 
and Environmental Sciences Institute (HES!) in 
Washington, D.C. Dr. Holsapple has published over 
1 50 manuscripts and chapters. After completing two 
years of postdoctoral work at the Medical College of 
VirginiaA/irginia Commonwealth University, he was 
appointed an Assistant Professor in the Department 
ofPharmacotogy and Toxicology. He was tenured and 
promoted to Associate Professor in 1 989. Dr. Holsapple 
served as the Director of his department's graduate 
program from 1987 until 1991, and he received the 
“Professor of the Year Award" in his department in 19B9. 
Dr. Holsapple joined the Toxicology, Environmental 
Research and Consulting Laboratories at the Dow 
Chemical Company in 1994 and was promoted to 
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Scientist in 2000, His responsibilities included serving 
as the Technical Leader of both the Immunotoxicology 
and the Respiratory Toxicology Groups. Dr. Holsapple 
left Dow in 2002 to join the HESI staff. Dr. Holsapple 
is currently an Adjunct Professor in the Department 
of Pharmacology and Toxicology at Michigan State 
University, He is a member of the American College of 
Toxicology and the Society ofToxicology (SOT). He is 
a charter member of the Immunotoxicology Specialty 
Section in the SOT, In recognition of his contributions 
to toxicology, Dr, Holsapple received the SOT 
Achievement Award in 1992. Dr, Holsapple became the 
Vice President-elect of SOT in 2008. 

KEVIN KNOBLOCH 

Kevin Knobloch is the President of the Union of 
Concerned Scientists (UCS). Knobloch first worked at 
UC5 from 1 989 to 1 992 as Legislative Director for Arms 
Control and National Security. He returned in January 
2000 and was named President in December 2003. He 
oversees the organization's research, public education, 
and legislative programs, Knobloch recently served as 
Chair of the Green Group, a coalition of the CEOs of 34 
national environmental organizations, and currently 
serves as Co-chair of the Green Group Climate and 
Energy Committee. He ied UCS delegations to the 
United Nations international Climate negotiations in 
Montreal in 2005 and in Bali in 2007. In addition to his 
positions at UCS, he served as Director of Conservation 
Programs for the Appalachian Mountain Club in 
Boston. During six years on Capitol Hill, he was the 
Legislative Director for U.5. Senator Timothy Wirth (D- 
CO) and Legislative Assistant and Press Secretary for U.S. 
Representative Ted Weiss (D-NY). He began his career 
as an award-winning newspaper journalist, writing 
for several Massachusetts publications. He recently 
completed eight years on the Board of Directors 
of the Coalition for Environmentally Responsible 
Economies and serves on the Environmental League of 


Massachusetts Board of Directors. He is also co-founder 
and former President of the Arlington (MA) Land Trust. 

KENNETH OLDEN 

Kenneth Olden has been the Founding and Acting 
Dean of the proposed School of Public Health at the 
City University of New York since 2008. Dr. Olden is 
a cell biologist and biochemist by training, and has 
been active in cancer research for over three decades. 
From 1 979 to 1 99 1 , Dr. Olden worked at Howa rd 
University in several roles, ultimately as Director of the 
Howard University Cancer Center and Chairman of the 
Department of Oncology. From 1 991 to 2005, Dr. Olden 
was Director of the National Institute of Environmental 
Health Sciences (N1EH5) and the National Toxicology 
Program, with a concurrent scientific post as Chief of 
the Metastasis Section of the NIEHS Environmental 
Carcinogenesis Program. Dr. Olden has maintained 
his research interests throughout his administrative 
career. Much of his work has focused on the role of 
glycoproteins in cancer. Working with Ken Yamada 
and others at the National Cancer Institute, he studied 
the glycoprotein fibronectin, and its possible role in 
inhibiting metastasis. 

ROGER A. PiELKEJR. 

Roger A. Pielke, Jr. has been on the faculty of the 
University of Colorado since 2001 and Is a Professor in 
the Environmental Studies Program and a Fellow of the 
Cooperative Institute for Research in Environmental 
Sciences (CIRES). At CIRES, Dr, Pielke served as the 
Director of the Center for Science and Technology 
Policy Research from 2001-07. His research focuses 
on the interseaion of science and technology and 
decision making. In 2006, Dr, Pieike received the Eduard 
Bruckner Prize in Munich, Germany for outstanding 
achievement in interdisciplinary climate research. 
Before joining the University of Colorado, from 1 993- 
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2001 , he was a Scientist at the National Center for 
Atmospheric Research. Dr. Pielke is an Associate Fellow 
of the James Martin Institute for Science and Civilization 
at Oxford University's Said Business School. He is also 
a 2008 Fellow of the Breakthrough Institute, He is also 
author, co-author or co-editor of five books. His most 
recent book is The Honest Broker: Makmg Sense of Science 
in Policy and Politics. 

SHERRI K. STUEWER 

Sherri Stuewer is Vice Prestdertt - Safety, Health and 
Environment for Exxon Mobil Corporation, in that 
role she is responsible for developing, reviewing, 
and coordinating Exxon Mobil’s worldwide efforts 
concerning the environment, safety, and health. 

Prior CO her current position, Stuewer was Strategic 
Planning Manager for Exxon Mobil, General Manager 
of the Exxon Company U.S.A. supply department, and 
Manager of the Exxon refinery in Baytown. Texas. Over 
her 33-year career with Exxon Mobil, she has held a 
variety of technical and managerial positions in refining, 
planning, and logistics. Stuewer is a member of the 
Board ofTrustees and the Engineering College Council 
at Cornell University. She is also a Board Member of the 
YMCA of Metropolitan Dallas and the Bermuda Institute 
of Ocean Sciences. She is a past Chair of the Industry 
Advisor Board to the international Energy Agency. 

WENDY E. WAGNER 

Wendy E, Wagner is the Joe A, Worsham Centennial 
Professor at the University of Texas School of Law 
and recently joined the Case Law School faculty as a 
Professor through a joint, half-time arrangement with 
the University of Texas. Prior to joining the University of 
Texas Law faculty, Wagner was a Professor at the Case 
Western Reserve University School of Law and School 
of Management, and was a Visiting Professor at the 
Columbia Law School and the Vanderbilt Law School. 


She writes primarily in the area of environmental 
law and science, exploring the ways that science is 
used and misused in decision-making by the courts, 
Congress, and the agencies. Wagner has participated 
as an officer or committee member in a number of 
professional societies, including several sections of the 
American Bar Association, the Society for Risk Analysis, 
the National Conference of Lawyers and Scientists, and 
has served on several National Academy of Sciences 
committees. Wagner began her legal career in 1 987, 
when she served as a law clerk for the Honorable Albert 
Engel, the Chief Judge of the U.S. Court of Appeals, 

Sixth Circuit, in Grand Rapids, Michigan. She then 
served as an Honors Attorney at the Environmental 
Enforcement Section of the Environment Division 
at the Department of Justice in Washington, D.C 
Wagner then moved to the General Counsel Office of 
the Department of Agriculture (USDA) in 1 991 where 
she served as the Pollution Control Coordinator and 
established a central office, with six satellite legal 
offices, to manage and advise USDA agencies on 
compliance under the pollution control laws. 

Staff 

DAVID GOLDSTON 

David Goldston served as Chief of Staff of the House 
Committee on Science from 2001 through 2006, the 
culmination of more than 20 years on Capitol Hili 
working primarily on science policy and environmental 
policy. Since retiring from the Congressional staff, 
Goldston has been a Visiting Lecturer at Princeton 
University's Woodrow Wilson School of Public and 
International Affairs and at the Harvard University 
Center for the Environment. He writes a monthly 
column for Nature on science policy titled “Party of 
One." He serves on the National Academy of Sciences’ 
Aeronautics and Space Engineering Board and on 
a panel of the Academy's Committee on National 



317 


65 


Statistics, He Co-chaired ars American Physical Society 
study on energy efficiency and has served on panels 
producing reports under the auspices of the American 
Academy of Arts and Sciences and 0MB Watch. 


JOSH TRAPANI 

Josh Trapani joined the staff of the Bipartisan Policy 
Center in 2008, Previously, he was an American 
Association for the Advancement of Science (AAA5) 
Science & Technology Policy Fellow on the Policy 
Analysis staff within the Research & Development 
Deputy Area, U.S. Forest Service, where his work 
focused on climate change adaptation and mitigation. 
Prior to that. Dr, Trapani was the American Geophysical 
Union's Congressional Fellow, working for Senator 
Dianne Feinstein (D-CA) on public lands, climate 
change, and other science issues. Dr. Trapani also 
holds a Research Collaborator position with the 
Department of Paleobiology at the Smithsonian 
Institution. Trained as a geoscientist, his research took 
him to sites throughout the United States as well as to 
Coahuiia, Mexico and the Omo Vaiiey of Ethiopia. He 
has published a dozen peer-reviewed papers, as well as 
essays on science and policy. 
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Administrative Conference Recommendation 2013-3 


Science in the Administrative Process 

Adopted June 14, 2013 


Over the last three decades, several authorities made recommendations for improving 
transparency in the use of science^ in the administrative process.^ Partially in response to these 
recommendations, the executive branch and Congress have made a number of reforms to the 
scientific process undergirding agency decisionmaking. In 2009, President Obama issued a 
memorandum directing that, "[t]o the extent permitted by law, there should be transparency in 
the preparation, identification, and use of scientific and technological information in 
policymaking."^ "Each agency should [also] have appropriate rules and procedures to ensure 
the integrity of the scientific process within the agency.”'' The Office of Science and Technology 
Policy (OSTP) elaborated upon this memorandum in 2010, instructing agencies to 
"communicate scientific and technological findings by including a clear explication of underlying 


'The scope of this recommendation is limited to the "natural sciences" (e.g., chemistry, physics, medical science, 
geology, etc.), mathematics, statistics, computer science, and other allied fields. It is based upon a report that 
deals with agency research and decisionmaking related to the natural sciences. WENDY Wagner, Science in 
regulation: a Study of Agency Decisionmaking Approaches (Feb. 18, 2013), available at 

http://www.acus. gov/sites/default/files/documents/Science%20in%20Regulation_Final°/E20Report__2_18_13_0.pdf 


' See e.g. Nat'l Research Council, Review or the environmental Protection Agency's Draft IRIS Assessment of 
Formaldehyde (2011); Comm, on Risk Assessment of Hazardous Air pollutants, Nat'l Research council. Science and 
Judgment in Risk Assessment (1994); NaTl Research Council, risk Assessment in the Federal Government: Managing the 
Process (1983); Bipartisan Polict Ctr., Improving the Use of Soence in Regulatory PoLia 16, 41-42 (2009) [hereinafter 
"BPC Report"]; see also Ctr. for EFFEaivE Gov't, Advancing the Public Interest through Regulatory Reform: 
Recommendations for President-Elect Obama and the 111™ Congress 26, 34, 47 (2008). 

' Memorandum from the Admin, of Barack H. Obama for the Heads of Executive Departments and Agencies on 
Scientific Integrity, Daily Comp. Pres. Docs., 2009 DCPD No. 00137 (Mar. 9, 2009) [hereinafter "Obama Scientific 
integrity Memo"], available at http://www.gpo.gOv/fdsYs/pkg/DCPD-200900137/pdf/DCPD-200900137.pdf. 

"id. 
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assumptions; accurate contextualization of uncertainties; and a description of the probabilities 
associated with both optimistic and pessimistic projections."^ 

At base, these initiatives demand heightened transparency of agencies' use of science as 
a central means of ensuring the basic accountability of agency regulation. If an agency 
identifies the role that scientific information plays in its ultimate decision and explains how it 
ensured that its scientific analysis was rigorous, then the public has a basis against which it can 
evaluate both the scientific and policy judgments underlying the agency's decision. This 
transparency allows those outside the agency to assess whether the agency's policy decision 
comports with the authorizing law and the scientific record. A transparent decisionmaking 
process also advances other institutional and scientific goals, such as identifying promising 
areas for future research and serving as a bulwark against misuse of science for political ends.® 

Despite these important initiatives, a study commissioned by the Administrative 
Conference’ (and public meetings that considered questions it raised) revealed that agency 
decisionmaking processes would benefit from further improvements. Drawing on this learning, 
the recommendation offers several proposals for enhancing the transparency of agencies' use 
of science. At the same time, the Conference recognizes that agencies' abilities to implement 
this recommendation may be affected by resource limitations. 

First, the recommendation highlights a number of innovative practices undertaken by 
different federal agencies to enhance the transparency of their scientific decisionmaking 
processes. As a general matter, agencies should articulate the specific questions to be 
informed by scientific information, specify study designs for new research, and establish criteria 


^ Memorandum from John P. Holdren, Director of the Office of Science and Technology Policy, to the Heads of 
Executive Departments and Agencies on Scientific Integrity (Dec. 17, 2010), available at http://www.whitehouse. 
gov/sites/defauit/files/mlcrosltes/ostp/5cientific-integrity-memo-12172010,pdf. To effectuate this and a number 
of other responsibilities, agencies were asked to report back to OSTP on the actions taken to develop and 
implement their scientific integrity policies by April 2011. 

^ BPC Report, supra note 2, at 3. 

^ Wagner, supra note 1. 
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for weighing existing studies.® Agencies should identify scientific reports or data upon which 
they relied and material literature that they considered, but upon which they did not rely, to 
the extent practicable and permitted by iaw.^ Agencies should establish checkpoints (i.e., times 
for closing off consideration of additional research or debate prior to making a final regulatory 
decision) and policies for reopening that consideration. Agencies should also consider 
extending attribution to individual staff who participate in the preparation of scientific reports 
and taking other steps to promote robust debate among agency scientists.^° In addition, 
agencies should share best practices with other agencies and should recommend the removal 

® In so doing, agencies should endeavor to explain the relationship between scientific research and the policy 
decisions the research is intended to inform. Nat'l Research Council, Comm, on the Institutional Means for 
Assessment of Risks to Public Health, Risk Assessment in the Federal Government: Managing the Process 7 (1983). 

® See Administrative Conference of the United States, Recommendation 2011-1, Lego! Considerations in E- 
Rulemaking, H 4, 76 Fed. Reg. 48,789, 48,789 (Aug. 9, 2011); see also Exec. Order. No. 13,642, Making Open and 
Machine Readable the New Default for Government Information, 78 Fed. Reg. 28,111 (May 14, 2013); 
Memorandum from John P. Holdren, Director of the Office of Science and Technology Policy, to the Heads of 
Executive Departments and Agencies on Increasing Access to the Results of Federally Funded Research (Feb. 22, 
2013) (calling for agency plans to permit public access to research papers funded in whole or in part with federal 
monies). As a general matter, the agency should make publicly available any scientific literature it considered, 
including literature it reviewed but upon which it ultimately did not rely. For purposes of the recommendation, 
literature that an agency "considered” includes not only any study an agency official relied upon but also any study 
an agency official reviewed but ultimately determined not to rely upon (because it was deemed to be outside the 
scope of the scientific study at hand, was not considered sufficiently reliable, or was otherwise rejected by the 
agency official). Cf. Administrative Conference of the United States, Recommendation 2013-4, The Administrative 

Record in Informal Rulemaking, Fed. Reg. (providing a similar definition of "consider" in the context of the 

administrative record in informal rulemaking). If an agency official merely had access to a study but did not 
specifically analyze it to determine its relevance, that study has not been "considered" within the meaning of the 
recommendation for purposes of making such literature publicly available, 

in response to President Obama's call for agencies to develop "appropriate rules and procedures to ensure the 
integrity of the scientific process," Obama Scientific Integrity Memo, supra note 3, a number of agencies have 
promulgated integrity policies to promote open debate among agency scientists. See, e.g., Envtl. Prot. Agency, 
Scientific Integrity Policy (Feb. 2012), available at http://epa.gov/osa/pdfs/ 
epa_scientific_integrity_poiicy_20l20ll5.pdf; Food and Drug Admin., Scientific Integrity at FDA, FDA Staff Manual 
Guides, Volume IV-Agency Program Direghves 2 (2012), available at http://www.fda.gov/ 
ScienceResearch/AboutScienceResearchatFDA/ucm306446.htm; Nat'l Oceanic and Atmospheric Admin., Scientific 
Integrity (Dec. 7, 2011), available at http://www.corporateservlces.noaa.gov/ame5/administrative_orders/ 
chapter_202/202-735-D.pdf; Nuclear Regulatory Comm'n, Collaborative Work Environment Program, 
http://www.nrc.gOv/about-nrc/values.html#open (last updated May 4, 2012); see also Francesca T. Grifo, Federal 
Agency Scientific Integrity Policies: A Comparative Analysis (Mar. 2013), http://www.ucsusa.org/ 
assets/documents/scientificJntegrity/SI-policies-comparative-analysis.pdf. 
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of any legal impediments to promoting transparency in decisions in which science is an 
important element.^^ 

Second, the recommendation offers a series of proposals to bring greater congruity to 
the treatment of publicly and privately funded scientific research. Specifically, it encourages 
the disclosure of data underlying scientific research, including both privately funded and 
federally funded research, that an agency is considering {to the extent practicable and 
permitted by iaw).^^ Similarly, it recommends extending conflict of interest disclosure norms to 
private parties who submit studies used by an agency. 



RECOMMENDATION 


Suggested Agency Practices Regarding the Use of Science in the Administrative Process 

1. Explaining Agency Scientific Decisionmaking. Agencies should explain in 
proposed and final decision documents how they ensured rigorous review of the scientific 
information underlying each science-intensive regulatory project. This includes a statement of 
how each agency evaluated the scientific information used in its analysis; how the agency made 
that information available to reviewers and the public; how the analysis was reviewed by 
experts and interested parties; and how the agency ensured that the final decision was 
supported by the scientific record, 

2. Assuring Transparent Assessments. At an early stage in their decisionmaking 
processes, agencies should identify the specific policy questions that may be informed by 
science; describe the design of the assessments needed to characterize risks and inform policy 
decisions; and describe the criteria to be used in reviewing and weighing existing studies. 
When completed, assessments should: identify other appropriate analytical choices and explain 

See Wagner, supra note 1, at 135-38 (identifying a number of externa! legal impediments to promoting 
transparency, including short statutory deadlines, limits on dissemination of scientific studies, resource limitations, 
and caps on the number of discretionary advisory committees agencies can constitute). 

Legal restrictions that may limit agencies' ability to provide such disclosures include, among other things, 
protections for personal privacy, trade secrets, and confidential business information. 
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why they were not chosen; provide a synthesis of the available evidence and relevant literature 
guided by the assessment design or criteria; identify significant assumptions and choices of 
analytical techniques; provide a statement of remaining uncertainties; and discuss how 
different plausible choices might change the results of the assessment. Where possible, 
agencies should also explain the relationship between their scientific findings and the final 
policy choice. Agencies should strive to communicate this information in a manner that is clear 
to the general public. 

3. Disclosing Underlying Studies and Data. To the extent practicable and permitted 
by law and applicable policies, each agency should identify and make publicly available (on the 
agency website or some other widely available forum) references to the scientific literature, 
underlying data, models, and research results that it considered. In so doing, the agency should 
list all information upon which it relied in reaching its conclusions, as well as any information 
material to the scientific analysis that it considered but upon which it ultimately did not rely. 
Consistent with the limitations in the Information Quality Act (IQA) guidelines issued by the 
Office of Management and Budget and its own IQA guidelines, each agency should ensure that 
members of the public have access to the information necessary to reproduce or assess the 
agency's technical or scientific conclusions. 

4. Checkpoints and Explanations. Agencies should consider establishing explicit 
checkpoints for regulatory projects, defining both the conditions under which they intend to 
close their consideration of research or debate in order to reach a decision and when they 
might reopen that consideration, particularly in cases when they are not bound by judicially 
enforceable deadlines. In any case, agencies should explain their decisions to initiate, stop, or 
reopen consideration of research or debate. Such explanations should reference significant 
relevant ongoing research or other relevant factors. 

5. Identifying Future Projects. For science-intensive projects, agencies should 
identify specific types of future research that may be needed to reduce significant uncertainties 
in order to advance understanding of the issues. 
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S. Attribution for Agency Personnel. Agency personnel play an important role in 
producing their respective agencies' scientific analyses. Agencies should consider providing 
their personnel with some form of consensual attribution for reports or analyses to which they 
contribute in a significant way. If appropriate, such attributions should be made for personnel 
who contributed in a significant way to a technical or scientific report, including not only 
scientists but also economists, lawyers, and other contributors. Reviewers and other 
contributors could be identified by name and general contribution. 

7. Encouraging Debate. Agencies should encourage vigorous debate among agency 
scientists and should explore ways of incorporating the diversity of that debate in any resulting 
work product. Agency employees should be encouraged to publish their scientific work in the 
peer reviewed literature, provided that they follow applicable agency procedures and that 
confidential governmental deliberations are not compromised. Dissenting staff members 
should be protected from reprisals. 

8. Sharing of Agency Best Practices. Agencies should identify and publicize the 
innovations they have developed for transparently incorporating science into their regulatory 
decisions. OSTP, an interagency group headed by OSTP, or another body should consider 
occasionally convening agency representatives to discuss and share best practices. 

9. Addressing Legal Obstacles to Transparent Decisionmaking. Agencies should 
identify legal obstacles that may impede otherwise appropriate public access to the scientific 
information underlying agency analyses or that may prevent the agencies' development of 
scientifically robust decisionmaking processes. Agencies should recommend appropriate 
actions to eliminate such impediments, including revisions in existing law, to the Executive 
Office of the President. 

Agency Disclosures to Enhance the Transparency of Research 

10. Data Disclosure. To the extent practicable and In compliance with applicable 
legal restrictions, privileges, protections, and authorities, agencies should seek to provide 
disclosure of data underlying scientific research, including both privately and federally funded 
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research being considered by the agencies. Where practicable, such information should be 
disclosed in machine-readable format. Where such data are not subject to legal or other 
protections, and the data's owners nonetheless will not provide such access, agencies should 
note that fact and explain why they used the results if they chose to do so. Agencies should 
review their confidential business information policies to ensure that they include appropriate 
mechanisms to prevent over-claiming. 

11. Conflict of Interest Disclosure. Agencies should require conflict of interest 
disclosures on all scientific research submitted to inform an agency's licensing, regulatory, or 
other decisionmaking processes. This disclosure should be similar to the conflict of interest 
disclosure required by some scientific journals, such as that used by the International 
Committee of Medical Journal Editors. The regulatory conflict of interest disclosure should 
also, where permitted by law, identify whether the experimenter or author had the legal right 
without approval of the sponsor of the research to: design the research; collect the data; 
interpret the data; and author, publish or otherwise disseminate the resulting report or full 
dataset. To the extent that a party other than the principal investigator (e.g., the study sponsor 
or funder) had control over the design or publication of the study, agencies should disclose this 
fact and specify the nature of the control such an entity exercised. 
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Recruiting Researchers 
Through Facebook 

To the Editor: 

W e havi; read with considerable interest 
Dr. Richiardi’s communication re- 
garding tire use of Facebook to recruit par- 
ticipants in theNlNFEA study.' .As previ- 
ous initiatives report, Facebook provides a 
positive virtual environment in which 
study participants can be enrolled and fol- 
lowed.'’^ As a widely used online social 
network, Facebook may be usefii! for 
odter research purposes. We w'ould like to 
share our experience in recruiting re- 
searchers through this website. 

In June 2011, our Collaborative 
Working Group for the Research of 
Human Resources for Health, Rcd- 
LIRHUS (Grupo Colaborativo Latino- 
americano para la Invcstigacion de Recur- 
sos Humanos en Salud), designed a 
multiccnter study to explore the profile 
and professional expectations of L.atin 
American medical studenLs, with ques- 
tions on topics such as emigration inten- 
tion and primary-care labor perspective.s. 

Tliis study tvas conceived as a 
continent-wide evaluation, using a pilot 
tested'^ self-administered survey. To 
gather these data, we decided to start the 
fieldwork by enrolling researchers from 
various countries in Latin .America via 
Facebook. Given that medical students 
were the potential study subjects, we 
decided to involve them also as the local 
principal investigators. 

In October 2011, the project was 
approved by the Ethical Committee of 
the Institute Nacional de Salud del Peril. 
Wc started with a limited number of 
universities, but this situation was insuf- 
ficient to achieve our objective. We, 
therefore, implemented a new reemiting 
strategy by posting an invitation on the 
“wall” of the local organizations’ Face- 
book pages (by country or university) of 
the Medical Students’ Scientific Societ- 
ies (Sociedades Cientificas de Estudi- 
antes de Mcdicina)'* and similar groups, 
such as the International Federation of 
Medical Students’ .Associations. 

500 I www.epidem.com 


Letters 


Ov*CTalL 80 researches agreed to take 
part in this project represented 80 uni- 
versities from 15 Latin American <x)untrics. 
Approximately half of diese rcsearcheis 
were recruited by Facebook. Also, in Octo- 
ber 2011, w’e created a Facebook “Closed 
Group,” called Red-LiRHUS, to connect all 
participant rcsearcheis (including tliosc who 
were cwitacted using Facdxwk and those 
who were not). This group provided rc- 
spestses to commcMi questions and included 
95% of the participant researchers in their 
local universities. 

As our experience suggests, Face- 
book can be helpful in recruiting and 
communicating with a research team, 
even in a multinational context. 

Renee Pcreyra-EIias 
Percy Mayta-Trist^n 
School of Medicine 
Universidad Peruana de Ciencias Apiicadas 
Lima, Pen'i 
percy.mayta@upc.cdu.pe 
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Benefits of Publicly 
Available Data 

To the Editor: 

T he National Morbidity, Mortality, 
and Air Pollution Study (NMMAPS) 
was de.signed to examine the health ef- 
fects of air pollution in the United 
State.s, The primary question was 
whether particulate matter was respon- 
sible for the associations between air 
pollution and daily mortality. Secondary 
questions concerned measurement error 
in air pollution and mortality displace- 

Epidemiology 


ment.' Since then, NMMAPS has been 
used to answer many important ques- 
tions in environmental epidemiology. 

The data from 108 US cities for 
the years 1987--2000 comprise daily ob- 
servations of mortality counts in 3 age 
groups, air pollutants (including partic- 
ulate matter and ozone), and weather 
(including temperature). 

Tlie data were made publicly avail- 
able first via the Web (http://ww\v.ihapss. 
jhsph.edu/data/data.htm), and then via R." 
They were removed from public availabil- 
ity in 201 1 due to piivacy concerns. 

We used a systematic review to find 
peer-reviewed papers or reports that used 
the NMMAPS data. There were 67 papers 
or reports in total (see the eAppendix 
[http://iinks.lww.com,''EDE/A575] for the 
methods and a complete list). 'Fliirty-three 
publications (49%) were based on tlie 
publicly available data (Table). The most 
common application was methodology 
(33 publications). ITie first authors came 
from 5 countries. 

Sixty-seven publications arc a 
substantial research output from one 
study, reflecting the originality and 
scope of the data. The data have been 
used to answer questions on the health 
effects of air pollution and temperature 
in the United States, and to an-swer 
methodological questions. Such applica- 
tions also create benefits outside the 
United States, as new and refined meth- 
ods can be used in other countries. 

NMMAPS has been used to ex- 
amine deaths during heat waves and to 
predict future heat-wave deaths due to 
climate change. Examining climate 
change was not an original goal of 
NMMAPS, but these data (which 
cover a wide range of climates over a 
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TABLE. Number of NMMAPS 
Publications by Their Dependence on 
Public Availability, Application, and 
Country of First Author, Ordered by 
Frequency (n = 67) 

Publications 


No. (%r 

-Arising from NMMAPS' public availability?'’ 

No 

34 (5!) 

Yes 

33 (49) 

Application 


Methodology 

33 (49) 

Air pollution 

19(28) 

Temperature 

5(7) 

Air pollution and methodology 

5(7) 

Air pollution and temperature 

4(6) 

Air pollution and season 
Country of first author 

Ml) 

United Slates of America 

39 (58) 

Australia 

18(27) 

United Kingdom 

8(12) 

Italy 

Ml) 

Greece 

1 (1) 

“Percents do not alway.s add to 
rounding. 

""Yes” if none of the authors w 
NMMAPS team authors. 

KK»% because of 

cr« from the L’S 


long time) arc an excellent resource 
for investigating this critically impor- 
tant question. 

Research from NMMAPS ha.s 
been used by US Environmental Pro- 
tection Agency for regulatory impact 
statements on particulates^ and ozone, 
and by reports on climate change.** It 
has influenced policy outside the 
United States, being cited in a UK 
government report on particulate mat- 
ter'’ and a World Health Organization 
report on air pollution.*^ 

The easy availability of NMMAPS 
meant that researciiers could write papers 
witliout the rime and co.sts of data collec- 
tion and ethics applications. The study has 
benefited the research careers of statisti- 
cians and epidemiologists around the 
world. This creates a virtuous circle, as 
researchers are more likely to stay in the 
field, produce more research outputs, and 
win grants. If this pattern Ls to continue, 
more researchers need to make their data 
publicly available. 

US investigators who w'orked to 
make NMMAPS publicly available 

® 2012 Lippincott WiUiams ct Wilkins 


should be applauded. More formal re- 
wards for providing access to data arc 
difficult, which may be one reason 
why more data sets are not made freely 
available.^ 

A key obstacle to data-sharing is 
the ethics of sharing medical data, 
w-ith the major concern being w'hether 
risks outweigh benefits. The number 
and influence of publications from 
NMMAPS outweigh (in our opinion) 
data-security concerns, particularly as 
the NMMAPS health data arc aggre- 
gated and anonymized. Nonetheless, 
data-security concerns caused these 
data to be removed from public avail- 
ability in 2011 — a backward step for 
reproducible research.® Thanks to 
NMMAPS, the United States has an 
improved understanding of the health 
effects of air pollution and heat waves. 
Furthermore, the utility of these data 
is not diminishing — 9 papers were 
published in 2010, 7 in 2011, and 
additional papers are currently under 
review. 
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On Influencing 
Population Means 

To the Editor: 

P ossible temporal trends in sperm 
concentration at the population level 
have been used to discuss the role of 
environmental factors (including endo- 
crine disrupters) on male fertility. An 
assumption sometimes made* is that 
temporal trends in biologic parameters, 
such as sperm concentration, will parallel 
temporal trends in their risk factors. As we 
illustrate later in the text, this assumption 
is simplistic, outside the (probably rare) 
situation w'here a single common environ- 
mental factor has a major impact on the 
biologic parameter of interest. 
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February 10, 2014 


The Honorable Lamar Smith 
Chairman 

Science, Space, and Technology Committee 
House of Representatives 
Washington, DC 20515 

Re: Secret Science Reform Act of 2014 1H.R.40121 


Dear Chaiiman Smith: 


The Halogenated Solvents Industry Alliance, Inc. (“HSIA"), an association of producers 
and users of chlorinated solvents, supports your Committee’s efforts to stop ERA from proposing 
regulations and adopting risk assessments based on science that is not transparent or 
reproducible. I write to bring to your attention a particularly egregious example of such ERA 
action involving trichloroethylene (“TCE”), a solvent that is often found at contaminated waste 
sites as a result of its legacy use by the electronics and manufacturing industries and by the 
Armed Services to clean aircraft, tanks, and ships. 

In September 201 1, ERA issued its long-awaited “Toxicological Review of 
Trichloroethylene (CAS No. 79-01-6) in Support of Summary Information on the Integrated Risk 
Information System (IRIS).” This IRIS Assessment contains a reference concentration (“RfC”) 
of 0.0004 ppm (0.4 ppb or 2 pg/m’) for TCE, a value considered by ERA to be protective for all 
non-cancer critical effects. 

In 2012, ERA Region IX began issuing action levels of 9 Mg/m’ (commercial and 
industrial) and 2 ug/m’ (residential) for short-term exposure to TCE at sites under its jurisdiction 
such as the Middlefield-Ellis-Whisman Superfund Site in Mountain View, California.' Because 
these levels are intended to protect against fetal cardiac malformations, they have raised a great 
deal of public concern at the Mountain View site (home to a new Google campus) and other 
places where they have been applied. Indeed, other ERA regions have applied comparable action 
levels to require evacuation of Naval facilities and other buildings. Compliance is problematic, 
as 2 Mg/m’ is within the range of background concentrations of TCE in urban air. 


' A December 13, 2013 letter to Stephen Hill of the California Regional Water Quality Control Board 
outlines and reaffirms the Region IX position: 

htln:/ /vo.seniite.eoa-gov/r9/.sfund/r9.sfdocw . nsty3dc2 B 3e6c.Sd6Q56f882574260074l7a2 /a 9da00560(l74ea5688257c38 
007628 l2/S Fr LE/9353864l.Ddf/BP.A% 2 dReeion%2'09%20South%2(Hiav%20VI%2(')Letlei%20l2-3- 1 3.odf 
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The action levels are all based on the IRIS RfC, which in turn is derived largely from a 
study by Johnson et al. entitled Threshold of Trichloroethylene Contamination in Maternal 
Drinking Waters Affecting Fetal Heart Development in the Rat, Environmental Health 
Perspectives 111: 289-92 (March 2003). The prohlem is that this study was not conducted 
according to EPA’s Good Laboratory Practices (“GLP”) and is not reproducible, as noted in the 
August 2013 peer review of a recent EPA “TSCA Chemicals Work Plan” assessment of TCE 
which was highly critical of EPA’s reliance on Johnson et al/. “One of the fundamental tenets in 
science is the reliability and reproducibility of results of scientific investigations.”’ The peer- 
reviewers noted: 

• At least two GLP-compliant studies conducted under botli EPA and Organization 
for Economic Coordination and Development (“OECD”) guidelines have been unable to 
reproduce the effect seen by Johnson et al., despite the participation in one of the studies by 
Johnson herself. 

• The dose-response relationship reported in Johnson et al. for doses spanning an 
extreme range of experimental dose levels is considered by many to be improbable, and has not 
been replicated by any other laboratory. This has been the subject of a series of articles in the 
scientific press.’ 

• The congenital heart defect incidence in control animals in Johnson et al. was 86 
times the historical control incidence in the rat strain they used. 

As Caiifomia EPA noted in declining to rely upon Johnson et al.: “These results are also 
not consistent with earlier developmental and reproductive toxicological studies done outside 
this lab in mice, rats, and rabbits. The other studies did not find adverse effects on fertility or 
embryonic development, aside from those associated with maternal toxicity (Hardin et al., 
2004)."'' 


HSIA’s attempts to see the raw data which formed the basis of the Johnson paper have 
been unsuccessful. The data supplied to EPA by Johnson were inadequate to validate or refute 
the paper’s conclusions. A direct appeal to Johnson has not made the data available for public 
scrutiny. And a Freedom of Information Act request (pursuant to the Shelby Amendment) has 
been denied by the National Institutes of Health, and is now on appeal. 


^ htta://ww\v. scccorp.com/t ct 20 1 3/»clfs/0j)061.t TCE FfN.\L All%2 ( )Reviewe r. s%2()Com!TieTits.doc x 

* Hardin, B, et at.. Trichloroethylene and Cardiac Malformations, Environ, Health Perspect. 1 12: A607-8 
(2004): “Johnson et al. (2003) provided no rationale for designing their study with a concurrent control five times 
larger than the treatment groups, which leads us to ask whether the control group reported here is, in fact, a 
composite of controls from multiple, perhaps five, different studies.. The immediate impact of this large conh'ol 
group is that the very cardiac ‘abnormalities’ at the 1,5 ppm dose that did not differ significantly from controls in 
1993 become statistically significant in 2003.” 


■1 California EPA Public Health Goal for Trichloroethylene in Drinking Water (July 2009), at 2 1 . 
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In sum, the RfC in the 201 1 IRIS TCE assessment is based on a flawed and 
irreproducible study, but it is being given the effeet of law by EPA. The action levels developed 
by Region IX were referred to EPA headquarters, and the question of setting short-term TCE 
exposure limits has resulted in significant ongoing disagreement among the EPA regions, as 
documented in the enclosed Inside EPA report of December 31, 2013 and numerous other 
articles in the trade press.’ Thus, the proper interpretation and use of tliis non-GLP study in risk 
assessment is a question of the highest priority to EPA’s Superfund program. 

There is a scientifically supportable way for EPA to resolve the controversy caused by its 
reliance on the Johnson et at. study. HSIA has offered, under a voluntary testing agreement in 
place with the Agency for Toxic Substances and Disease Registry (“ATSDR”), to fund a 
definitive study on the relationship between TCE and cardiac malformations that would be 
designed by a panel including representatives of ATSDR and EPA. Our letter of April 23, 2013 
on this subject is enclosed. Altliough such a study, if accepted, could have been conducted and 
reported by now, neither ATSDR nor EPA has responded to our offer. 

I hope this is helpful to your Committee. Please do not hesitate to let me know if we can 
provide further input. 


Very truly yours, 




Faye Graul 
Executive Director 


Enclosures 


» See, e.g., DOD Uses New TSCA Assessment to Criticize Trichloroethylene IRIS Value, Inside EPA (June 
3, 20 1 3); Exposure Uncertainties May Hamper EPA Effort To Assess TCE’s Risks, Inside EPA (April 25, 20 1 3); 
Amidst Review, EPA Scientists Defend Finding on TCE’s Heart-Defect Risks, Inside EPA (February 15, 2013); 
Massachusetts Adds to Scrutiny of EPA TCE Risk Assessment’s Adequacy, Inside EPA (February 1 1, 2013); New 
Jersey Short-Term TCE Limits Add to Growing Array of Approaches, Inside EPA (February 6, 2013); Regions Split 
Over Short-Term TCE Limit, Highlighting Need for EPA Guide, Region X TCE Guidance, Inside EPA (January 2, 
2013). 


4837-6515-25.W.1. 
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INSIDEEPA.COM 

an online news service from the publishers of IhsWe B’A 

Risk Policy Report - 12/31/2013 

Region IX Short-Term Limits For TCE Exposure May 
Guide National Policy 


Posted: December 30, 2013 

EPA Region iX is urging California state regulators to implement strict action levels and strengthened sampling strategies at certain 
Superfund sites in the San Francisco Bay area to protect against short-term exposures to the solvent trichloroethylene (TCE), 
measures that likely indicate how EPA will address TCE*s short-term risks nationwide, according to an inforrned source. 

Since late 2012, EPA officials have said agency headquarters is seeking a consistent nationwide policy for assessing and mitigating 
the risk of fetal cardiac malformations, a novel risk from short-term in utero exposures to TCE. The agency has yet to produce a 
document for dealing with the risk calculation for the birth defect, which first appeared in the September 201 1 Integrated Risk 
Information System (IRIS) TCE assessment. Since then, states. EPA regional officials and industry have been struggling to account 
for it in cleanup plans at contaminated sites, 

In a Dec. 3 letter, EPA Region IX urges the California Regional Water Quality Control Board (CRWQCB) to follow a program for 
assessing risks from short-term exposures to TCE that includes stringent prompt response action levels. The letter also asks the 
Golden State regulators to adopt sampling strategies consistent with EPA's recent draft guidance for assessing and mitigating vapor 
intrusion risks from chlorinated solvents. Vapor intrusion occurs when toxic vapors rise into buildings from underground 
contamination. The letter is available on lnsideEPA.com. {Doc ID: 2456609) 

In the letter, Kathleen Salyer, assistant director of the Superfund Division of Region IX's California Site Cleanup Branch, 
recommends mitigation measures for when prompt response action levels are exceeded. The limits and stricter sampling strategies 
are intended to reduce in utero exposures at nine sites contaminated with chlorinated solvents. 

In an attachment to the Dec. 3 letter. Region IX urges California water quality regulators to adopt a prompt response action level of 
9 micrograms per cubic meter (ug/m''3) to protect workers at commercial and industrial facilities during an 8-hour work day, and of 7 
ug/m''3 for a 1 0-hour workday, Additionally, Region IX recommends a prompt action level of 2 ug/m^3 to protect against residential 
exposures. 

When indoor air limits are reached, the memo recommends interim measures to mitigate short-term risks. These measures include 
fans or ventilation to increase building pressurization, installing subslab and or crawlspace depressurization systems, or a soil vapor 
extraction system. Region IX also suggests evacuation to eliminate exposures, especially when immediate response levels are 
exceeded, 

"In reviewing the multiple lines of evidence that have been collected for the SouUi Bay Sites, EPA Region 9 has identified data gaps 
that must be filled to fuily evaluate the potential for vapor intrusion into buildings overlying the South Bay Sites contamination," 
Saiyer writes to Stephen Hilt, Chief of CRWQCB's Toxics Cleanup Division. 

A source familiar with the thinking of some EPA officials says the Region IX plan could prompt EPA headquarters to issue a very 
similar policy for assessing and mitigating risks to pregnant women from short-term exposures to TCE in the near future. 

"This may force the hand of headquarters to get something out’ to address risks from short-term exposures to TCE, the source 
says, "it's really a sign, not just for the levels, but for sampling pracUces, that this is a direction where EPA as a whole is heading." 

In addition, the source says headquarters has not objected to Region IX's approach, and that the delay in creating a nationwide 
policy to protect against short-term exposures to TCE likely stems from debate over whether EPA's Office of Solid Waste and 
Emergency Response (OSWER), which recently crafted draftvaporintruson guidance, or the agency's Office of Research and 


http://irtsideepa.com/Risk-PoIicy-Report/Risk-Policy-Report-12/31/2013/region-ix-short-ter... 1/2/2014 
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Development, which includes the IRIS program that addresses chemical-spedfic challenges, should issue the policy. 

The substance of a future national policy wilt likely be similar to those ouUined in the recent Region IX memo and the limit Region X 
set to protect against short-term exposures late last year, the source says. ”1 haven't heard anyone at EPA questioning the genera! 
approach that Region IX or Region X have taken with setting numbers." 

In response to a request for an interview or comment on the Region IX letter, an EPA spokeswoman said the agency is working with 
its regional officials to address vapor intrusion risks at sites contaminated with TCE and that specific action levels to protect against 
short-term exposures are not part of the OSWER guidance for assessing and mitigating vapor intrusion from chlorinated solvents. 

Since the publication of the IRIS assessment for TCE, the Defense Department (DOD) and industry officials have criticized the 
science supporting the fetal cardiac risk as too uncertain for use in regulation, and said limits derived to protect against short-term 
exposures are based on conservative assumptions. In comments submitted on EPA's vapor intrusion guidance, DOD has urged the 
agency to create "defensible procedures" for setting limits to protect against short-term exposures and for responding to them at 
military facilities. 

EPA R^ion IX has been a leader in efforts to protect against the birth defects risk at contaminated sites, and in early 201 2 
proposed a removal action level (RAL) of 15 ug/m*3 to protect workers at the Middlefield-Ellis-Whisman (MEW) Superfund site in 
Mountain View, CA, from short-term exposures to TCE. That limit as well as the recent prompt response action levels are derived 
from the IRIS assessment for TCE. which set a reference concentration (RfC) - the amount of a substance EPA believes can be 
inhaled daily over a lifetime without adverse effects - of 2 ug/m^S. 

The RfC, which protects against chronic exposures, was based in part on a 2003 toxicology study by Paula D. Johnson that showed 
fetal cardiac malformations in lab rats exposed to TCE. EPA said the study indicated the birth defects could occur from inhaling the 
substance during pregnancy, implying risk from short-term exposure. Several peer review panels backed EPA's use of the study 
and the birth defects risk. 

Region IX's new prompt action levels are based on a hazard quotient (HQ) of 1. though the letter states that similar levels indicating 
a need for an immediate response may be derived using an HQ of 3. per a 2IX>8 OSWER policy memo for calculating RALs. The 
HQ represents the ratio of the exposure level to a calculated 'safe' dose. 

The levels are similar to Region IX's proposed 2012 RAL for the MEW site of 15 ug/m*3, using an HQ of 3. The Region IX prompt 
response action level is also similar to a level that Region X recommended last year of B.4 ug/m'-S In indoor air at commercial and 
industrial sites, and identical to Region X's limit of 2 ug/m^3 to protect against exposures in homes. 

But in a Dec. 13, 201 2, memo, Region X described its limits as "not-to-be-exceeded concentrations, as averaged over any 21-day 
period of time," noting that current science is unclear on whether shorter spites in indoor air contamination during a 21-day period 
cause birth defects. 

Region IX limits use a different exposure duration than Region X, as the RAL assumes a single daily exposure above 15 ug/m''3 for 
pregnant women couid result in fetal cardiac defects, while the prompt response action levels are averaged over an eight or 10-hour 
workday. 

The source Says the question of exposure duration remains a challenge for EPA in part because current technology that can 
measure variability in indoor air levels over extended durations is not cost-effective. In the Dec, 3 memo. Region IX requires 
sampling practices included in OSWER's draft vapor intrusion guidance for assessing and mitigating risks from vapor intrusion that 
was released for public comment in April, saying the document's multiple lines of evidence approach must be followed "to fully 
evaluate the potential for vapor intrusion into buildings overiying" the nine contaminated sites. 

Region IX requires multiple rounds of sampling from mulOple locations, and also says sampling should be conducted in crawlspaces 
and basements. In addition, the letter calls for sampling in cold weather months when some sites contaminated with TCE in the San 
Francisco Bay area have been found to have contamination levels in indoor air two to three times higher than in other seasons. 

A state regulator criticized some assessment approaches backed in the letter, saying recommendations including sampling when 
the air conditioning systems are not running are not technically defensible or are excessively conservative and will lead to 
unnecessary precautions at some homes and buildings. The source declined to comment on the response action levels for TCE, 
citing scientific uncertainty surrounding that issue. — Dave Reynolds 

2456859 


http://insideepa.com/Risk-Policy-Report/Risk-Poncy-Report-I2/31/2013/region-ix-short-ter,., 1/2/2014 
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April 23,2013 

Dr. Edward Murray 
Acting Director 

Division of Toxicology and Human Health Sciences 
Agency for Toxic Substances & Disease Registiy 
1600 Clifton RoadNE, MS F-57 
Atlanta, GA 30333 

Re: Proposal for Trichloroethylene tTCEt Developmental Toxicity Study 
Dear Dr. Murray: 

When we met with you in October regarding the protocol for the developmental 
neurotoxicity study that the Haiogenated Solvents Industry Alliance, Inc. (HSIA) is 
conducting pursuant to its agreement with the Agency for Toxic Substances and Disease 
Registry (ATSDR) to fill certain priority data needs identified by ATSDR, there was some 
discussion as to whether it might be possible to expand that study to include a developmental 
toxicity component that would address whether and to what extent TCE might play a role in 
causing cardiac anomalies in pups of exposed animals. As you are aware, this is a subject of 
some controversy in the scientific literature, with serious regulatory implications for the 
evaluation of short-term human exposures arising from vapor intrusion at contaminated sites. 

There was a consensus among the scientists we consulted that expanding the 
developmental neurotoxicity study would not be practical. Given our shared interest in 
clarifying whether TCE is a developmental toxicant. We approached you to see if ATSDR 
would have an interest in expanding our agreement to include a study intended to address the 
cardiac endpoint. During a conference call on March 6, it was agreed that we would provide 
you a description or skeleton protocol for the kind of study we have in mind to see whether 
ATSDR and the other Tri-agency Superfund Applied Research Committee (TASARC) 
members believe that such a study could help resolve this important issue. 

1. Rationale for Study 

Johnson et al. (2003) reported cardiac effects in rats from research earned out at the 
University of Arizona and originally published ten years earlier by the same authors.' In the 
earlier-published study, there was no difference in the percentage of cardiac abnormalities in 


‘ Johnson, P, et al.. Threshold of trichloroethylene contamination in maternal drinking waters affecting fetal heart 
development in the rat, Environ. Health Perspect. Ill: 289-92 (2003). Dawson, B, el a!., Cardiac teratogenesis of 
haiogenated hydrocarbon-contaminated drinking water, J. Am. Coif. Cardiol. 21: 1466-72 (1993). 
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rats dosed during both pre-mating and pregnancy at drinking water exposures of 1100 ppm 
(9.2%) and 1 .5 ppm (8.2%), even though there was a 733-fold difference in the 
concentrations. The authors reported that the effects seen at these exposures were 
statistically higher than tire percent abnormalities in controls (3%i). For animals dosed only 
during pregnancy, the abnormalities in rats dosed at 1100 ppm (10.4%) were statistically 
higher than at 1.5 ppm (5.5%), but those dosed at 1 .5 ppm were not statistically different 
from the controls. Thus, no meaningful dose-response relationship was observed in either 
treatment group. 

In 2003, Johnson etal. republished data from the 1.5 and 1100 ppm dose groups 
published by Dawson et al. in 1993 and pooled control data from other studies, an 
inappropriate statistical practice. When compared against the pooled control group, 
statistically significant increased rates of abnormalities were reported for the 250 ppb and 
1 100 ppm groups but not for the intermediate 1 .5 ppm group. The authors concluded that 
rats exposed to levels of TCE greater than 250 ppb during pregnancy have increased 
incidences of cardiac malformations in their fetuses. 

Johnson el al has been heavily criticized in the published literature , 2 and the 
University of Arizona studies were expressly rejected as the basis for a public health goal 
(PHG) by the California Office of Environmental Health Hazard Assessment (OEHHA)^ and 
minimal risk levels (MRLs) by the ATSDR.* Moreover, the Johnson et al findings have not 
been reproduced by any other investigators. In one study conducted by Fisher et al, 


2 Hardin, B, et al.. Trichloroethylene and cardiac malformations. Environ. Health Perspecl. 1 12: A607-8 (2004); 
Watson, R., et ai, Trichloroethylene-contaminated drinking water and congenital heart defects: a critical analysis of 
the literature, Repro. Toxicol. 21: 117-47 (2006). 

"Johnson el ai (2003) reported a dose-related increased incidence of abnormal hearts in offspring of Sprague 
Dawley rats treated during pregnancy with 0, 2.5 ppb, 250 ppb, 1.5 ppm, and 1,100 ppm TCE in drinking water (0, 
0.00045, 0.048, 0.218, and 128.52 mg/kg-day, respectively). The NOAEL for the Johnson study was reported to be 
2.5 ppb (0.00045 nig/kg-day) in this short exposure (22 days) study. The percentage of abnormal hearts in the 
control group was 2.2 percent, and in the treated groups was 0 percent (low dose), 4.5 percent (mid dose 1), 5.0 
percent (mid dose 2), and 10.5 percent (high dose). The number of litters with fetuses with abnormal hearts was 16.4 
percent, 0 percent, 44 percent, 38 percent, and 67 percent for the control, low, mid 1 , mid 2, and high dose, 
respectively. The reported NOAEL is separated by 100-fold from the next higher dose level. The data for this study 
were not used to calculate a public-health protective concentration since a meaningful or interpretable dose-response 
relationship was not observed. These results are also not consistent with earlier developmental and reproductive 
toxicological studies done outside this lab in mice, rats, and rabbits: The other studies did not find adverse effects on 
fertility or embryonic development, aside from those associated with maternal toxicity (Hardin el ai, 2004)." 
California EPA Public Health Goal for Trichloroethylene in Drinking Water (July 2009), p. 21. 

‘‘ ATSDR concluded that “[t]he study is limited in that only two widely spaced exposure concentrations were used 
and that a significant dose-response was not observed for several exposure scenarios.” Toxicological Profile for 
Trichioroethyiene Update (September 1997), p, 88. More recently, however, following publication by EPA in 2010 
of its TCE IRIS Assessment, ATSDR issued an Addendum that bases both chronic and intermediate-duration MRLs 
on the EPA RfD/RfC values (0.0005 mg/kg/day /0.0004 ppm (2 wg/m^)), which in turn are based in part on Johnson 
ei al. Addendum to Toxicological Profile for Trichloroethylene (January 2013). 



Dr. Edward Murray 
April 23,2013 
Page 3 


342 


pregnant rats were administered a daily dose of 500 mg TCE/kg by gavage on gestational 
days 6 through 1 5 and despite employing air improved method for assessing cardiac defects 
and the participation of Johirson herself in the study, the heart malformation incidence for 
fetuses from TCE-treated dams did not differ from control values on either a per fetus or per 
litter basis. 5 

No increase in cardiac malformations was observed in a second guideline, GLP- 
quality inhalation study reported by Carney et al.,^ despite high exposure concentrations (i.e,, 
0, 50, 150 and 600 ppm TCE for 6 hours/day and 7 days/week from gestational days 6 
through 20) and techniques capable of detecting most of the malformation types reported by 
Johnson etal. Carney et al. concluded that the no observed effect concentration (NOEC) for 
fetal effects was 600 ppm. Carney et al. is the published report of the developmental toxicity 
study HSIA sponsored under our voluntary agreement with ATSDR to fill certain priority 
data needs for TCE. 

In summary, based on a critical evaluation of these studies, the dose-response 
relationship reported in Johnson et al. for doses spanning an extreme range of experimental 
dose levels is considered by many to be improbable.^ Johnson et al. remains a poor basis for 
assigning human hazard, and no mechanistic or other studies provide information that 
bridges the gap to support the Jolinson et al conclusions. 

2. Proposal 

We propose to sponsor a definitive study to confinn the developmental effects of 
TCE in rats claimed by Jolinson et al. and subsequently used by EPA in its 201 1 
development of an RfD and RfC. As a GLP inhalation developmental toxicology study has 
already been conducted under a protocol approved by TASARC, it would seem most 
appropriate that the proposed study focus on oral ingestion, as did Johnson et al. There are 
many variables associated with study design (i.e., dosing range, dosing vehicle, dissection 
methods, etc.,) so the actual protocol for this study should be adopted by consensus under the 
guidance of an expert panel, which would include an author from each of the earlier, 
conflicting studies as well as recognized academic or consulting experts in developmental 
toxicity. In order for the study to have credibility with the regulatory community, ATSDR 
and the other TASARC agencies would need to be represented on the panel and approve the 
protocol. 


’ Fisher, J, el al, Trichloroethylene, trichloroacetic acid, and dichJoroacetic acid; do they affect fetal rat heart 
development? Int, J. Toxicol. 20: 257-67 (2001). 

Carney, E, et al. Developmental toxicity studies in CrhCd (SD) rats following inhalation exposure to 
trichloroethylene and perchloroethylene, Birth Defects Research (Part B) 77: 405--412 (2006). 


^ “Johnson and Dawson, with their collaborators, are alone in reporting that TCE is a ‘specific’ cardiac teratogen.” 
Hardin, B, et al. Trichloroethylene and cardiac malformations. Environ. Health Perspect. 112; A607-8 (2004). 
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Under the guidance of the panel, the proposed study would be put out for bid to 
reputable test laboratories with the appropriate expertise in developmental toxicity. If dose- 
related cardiac malformations are, in fact, observed, agency reliance on the University of 
Arizona studies would be confirmed. However, if the Johnson et al. results are not 
confirmed, it would be accepted that those studies should not be used as the basis for the 
RfD, RfC, MRLs, or other such guidance values for TCE. HSIA would expect that the 
results from the existing inhalation study and the proposed oral study, both conducted as part 
of ATSDR’s voluntary program to fill priority data needs, would be accepted as the 
appropriate basis for the development of inhalation and oral non-cancer hazard values. 

HSIA appreciates being given the opportunity to contribute to the resolution of this 
controversial issue and hopes that the members of TASARC share our commitment to 
application of the best science to the development of regulatory decisions. We look forward 
to hearing your thoughts on this proposal and would certainly be available to discuss it in 
greater detail at your convenience. I can be reached by phone at either (703) 875-0684 
(office) or (202) 286-6464 (cell) or by email atjbell@hsia.org. 


Sincerely, 

/ I ‘>11 

John U. Bell, Ph.D., DABT 
Director of Scientific Programs 


o 



